[ovs-dev] [PATCH/RFC v18 5/6] Add packet recirculation

Simon Horman horms at verge.net.au
Wed Feb 12 08:05:11 UTC 2014


Recirculation is a technique to allow a frame to re-enter
frame processing. This is intended to be used after actions
have been applied to the frame with modify the frame in
some way that makes it possible for richer processing to occur.

An example is and indeed targeted use case is MPLS. If an MPLS frame has an
mpls_pop action applied with the IPv4 ethernet type then it becomes
possible to decode the IPv4 portion of the frame. This may be used to
construct a facet that modifies the IPv4 portion of the frame. This is not
possible prior to the mpls_pop action as the contents of the frame after
the MPLS stack is not known to be IPv4.

Design:
* New recirculation action.

  ovs-vswitchd adds a recirculation action to the end of a list of
  datapath actions for a flow when the actions are truncated because
  insufficient flow match information is available to add the next
  OpenFlow action.  The recirculation action is preceded by an action
  to set the skb_mark to an id which can be used to scope a facet lookup
  of a recirculated packet.

  e.g.  pop_mpls(0x0800),dec_ttl becomes pop_mpls(0x800),set(skb_mark(id)),recirculate

* Datapath behaviour

  Then the datapath encounters a recirculate action it:
  + Recalculates the flow key based on the packet
    which will typically have been modified by previous actions
  + As the recirculate action is preceded by a set(skb_mark(id)) action,
    the new match key will now include skb_mark=id.
  + Performs a lookup using the new match key
  + Processes the packet if a facet matches the key or;
  + Makes an upcall if necessary

* No facet behaviour

  + Loop:
    1) translate actions
    2) If there is a recirculate action, execute packet
       and go back to 1) for remaining actions.

Co-Authored-by: Joe Stringer <joe at wand.net.nz>
Signed-off-by: Simon Horman <horms at verge.net.au>

---

Change Log:

v18
* Rework to work without facets present in user-space

v17
* Rebase for:
  - Removal of tag library
  - hiding of Hide rule_dpif_miss_rule()
* Do not add parent_facet variable to facet_remove()
* Recirculate if a vlan action occurs after a pop_mpls action.
  - It is possible for a VLAN tag to appear after an MPLS action
    and for a VLAN packet to result from an MPLS pop action.
    However, in this case the flow does not have any VLAN information -
    as the packet didn't look like a VLAN packet when it was an MPLS packet -
    and thus translation of VLAN actions does not work as expected.
    This may be mitigated by using recirculation to re-evaluate
    the flow of the packet after MPLS pop occurs.

v16
* Ensure that locks are released and memory freed in the case
  of recirculation in dpif_netdev_execute()
* Separate controller and MPLS tests
  - MPLS adds a number of tests and there are already a number
    of controller tests. It seems to make sense to separate them.
* Do not add reference count to facet
  - This does not appear to be necessary any more as situations
    where the parent-facet of a recirculated child-facet does not exist
    may be handled using the drop rule.
* Do not add 'strict' parameter from get_facet_chain.
  This is no longer useful as recirculated facets chains
  may me incomplete if parent-facets may be removed before child-facets.
  This is quite possible now that there is no reference count.
* Clarify comments relating to recirculated facet chain revalidation

v15
* Rebase for ofproto-dpif-xlate modularization
* Replace rule pointers in facet with offsets
 - Use rule offset to determine if facet was created by recirculation
 - Refactor xin->ofpacts initialisation to handle cases where the rule
   has disappeared but the facets still exist

v14
* Rename execute_recirculate() as commit_recirculate(),
  this seems more consistent with the naming of related functions
* Allow matching of zero skb_mark
 - Include skb_mark in flow key passed from datapath
 - Include skb_mark in flow key constructed from flow if the mask is non-zero
* Set skb_mark mask for recirculated facets during revalidation
* Run through facets in reverse order in facet_revalidate(), so that children
  are always destroyed before their parents.

v13
* Rebase for megaflow changes: very extensive rebase

v12
* Rebase
* Copy flow tunnel when recirculating in xlate_and_recirculate().
  This avoids a bug that occurred when recirculating more than once
  and thus using flow_storage as the flow passed to flow_extract()
  and flow_storage.tunnel as the tunnel passed to flow_extract().
  The tunnel parameter passed to flow_extract() must not be
  the tunnel element of its flow parameter.
* In xlate_and_recirculate() ensure that there is sufficient headroom in
  packet for both the key, which is present if xlate_and_recirculate()
  is called by handle_flow_miss_without_facet(), and an MPLS LSE in case
  mpls_push() occurs.

v11
* As suggested by Jarno Rajahalem
  - Rename xlate_with_recirculate() as xlate_and_recirculate and add
    a comment above it to make it clearer that it may modify its
    'packet' argument.
  - Only hash in facet_find_by_id() if it is needed.
  - Replace may_recirculate with may_xlate_l3_actions in do_xlate_actions()
    as this seems to be a more intuitive name
  - Fix spelling of execute_recirculate_action()
  - Remove spurious comment regarding setting of the recirculation_id element
    of struct flow in action_xlate_ctx_init() - there is no such element
    at this time
* Enhance comment for recirculation_id element of struct action_xlate_ctx
  to describe valid initial values.
* Enhance comments regarding the use of recirculation ids in facet_revalidate()
* Correct comment for recirculated element of struct action_xlate_ctx,
  previously its described states were the opposite of the implementation.
* Add comments above peek_recirculation_id() and get_recirculation_id()
* Correct error in parameter passed to odp_execute_actions()
  in xlate_and_recirculate()

v10
* Rebase
* Remove execute_actions_for_recircualtion() helper and
  use odp_execute_actions directly()

v9
* Rebase

v8
* For a learn action recirculate regardless of the value of
  ctx->may_learn in do_xlate_actions(). Previously recirculation
  only occurred if ctx->may_learn was true. That is a bug as
  it causes revalidation to fail. Flagged by the tests developed
  as part of a patch to allow multiple MPLS push and poop actions.
* Update tests removing unnecessary dl_type pre-requisite

v7
* Rebase
* Add skb_priority, skb_mark and tun_key parameters to dp_netdev_port_input()
  This allows the skb_priority, skb_mark and tun_key  to be passed from
  dpif_netdev_execute() to honour corresponding set actions.
* Recirculate on reg load and move, stack push and pop,
  multipath, bundle, and learn actions.
* Add tests

v6
* As suggested by Jesse Gross
  - Enforce MAX_RECIRCULATION_DEPTH rather than MAX_RECIRCULATION_DEPTH + 1
    by pre-decrementing rather than post-decrementing loop counter.
  - Stop processing actions when a recirculation action is met
    in validate_and_copy_actions__(). This reflects how
    actions are actually executed and will invalidate actions
    if anything follows recirculation.
  - Add a rate-limited warning for packets dropped because they
    are recirculated too much in ovs_dp_process_received_packet().
  - Pass skb_mark to dp_netdev_sample() rather than using a
    private one.
  - Handle skb_priority when executing recirculation without
    facets in ovs-vswitchd
  - Handle tunnel key when executing recirculation without
    facets in ovs-vswitchd
  - Do not duplicate counting of flow statistics for recirculated packets.
  - Handle revalidation of facets of revalidated packets
    - Also, add facet_lookup_valid_by_id() and use it when
      looking up the parent facet of a recirculated packet instead
      of using the facet without regard for weather it is valid or not.
* Merge with patch "Allow recirculation without facets"
* Merge with patch "Avoid recirculation id collision"

v5
* Correct declaration of facet_find_by_id to match definition:
  ovs_be32 -> uint32_t.

* Enhancements to recirculation id code:
  - Allow efficient lookup of facets by their recirculation id
  - Add RECIRCULATION_ID_DUMMY which may be used in cases
    where no facet it used. It is an arbitrary valid id.
  - Also add recirculated element to action_xlate_ctx()
    to use to detect if a recirculation action was added during
    translation. The previous scheme of checking if recirculation_id
    was not RECIRCULATION_ID_NONE is broken for cases where
    the context is initialised with a recirculation_id other than
    RECIRCULATION_ID_NONE. E.g. when RECIRCULATION_ID_DUMMY is used.

* Avoid recirculation id collision

  Avoid recirculation id collision by checking that an id is
  not already associated with a facet.

  Consecutive recirculation ids are used and thus it is possible for
  there to be situations where a very large number of ids have to
  be checked before finding one that is not already associated with a facet.

  To mitigate the performance impact of such situations a limit on
  the number of checks is in place and if no unused recirculation id
  can be found then the miss is handled without facets as this can
  be done using a dummy recirculation id.

rfc4
* Minor enhancement to recirculation id management: Add RECIRCULATE_ID_NONE
  to use instead of using 0 directly.

* Correct calculation of facet->recirculation_ofpacts and
  facet->recirculation_ofpacts_len in subfacet_make_actions()
  in the case of more than one level of recirculation.

* Allow recirculation without facets

  This covers the following cases:
  - Handle flow miss without facet
    + Previously the use of facets was forced if there was
      any chance of a recirculation action. That is, for
      all flows misses of MPLS packets.
  - Packet Out

rfc3

* Use IS_ERR_OR_NULL()

* Handle facet consistency checking by constructing a chain of facets
  from the given facet, to its recirculation parent and then its parent
  until the topmost facet.  If there is no recirculation  the chain will
  be of length one. If there is one recirculation action then the chain
  will be of length two. And so on.

  The topmost facet in the chain can is used to lookup the rule to be
  verified. The chain is then walked from top to bottom, translating
  actions up to the end or the first recirculation action that is
  encountered, whichever comes first. As the code walks down the chain
  it updates the actions that are executed to start of the actions to
  be executed to be just after the end of the actions executed in the
  previous facet in the chain. This is similar to the way that facets
  are created when a recirculation action is encountered.

rfc2
* As suggested by Jesse Gross
  - Update for changes to ovs_dp_process_received_packet()
    to no longer check if OVS_CB(skb)->flow is pre-initialised.
  - Do not add spurious printk debugging to ovs_execute_actions()
  - Do not add spurious debugging messages to commit_set_nw_action()
  - Correct typo in comment above commit_odp_actions().
  - Do not execute recirculation in ovs-vswitchd, rather allow
    the datapath to make an upcall when a recirculation action
    is encountered on execute.
    + This implicitly breaks support for recirculation without facets,
      so for now force all misses of MPLS frames to be handled with
      a facet; and treat handling of recirculation for packet_out as
      a todo item.
  - Use skb_mark for recirculation_id in match. This avoids
    both expanding the match and including a recirculation_id parameter
    with the recirculation action: set_skb_mark should be used before
    the recirculation action.
  - Tidy up ownership of skb in ovs_execute_actions

rfc1
* Initial post
---
 datapath/actions.c                   |   9 +-
 datapath/datapath.c                  | 100 +++---
 datapath/datapath.h                  |   2 +-
 datapath/flow_netlink.c              |   9 +
 include/linux/openvswitch.h          |   4 +
 lib/dpif-netdev.c                    |  49 ++-
 lib/dpif.c                           |   1 +
 lib/flow.h                           |   3 +
 lib/odp-execute.c                    |  32 +-
 lib/odp-execute.h                    |   2 +-
 lib/odp-util.c                       |  15 +-
 lib/odp-util.h                       |   1 +
 lib/ofp-actions.c                    |   6 -
 lib/ofp-actions.h                    |   6 +
 ofproto/automake.mk                  |   2 +
 ofproto/ofproto-dpif-recirculation.c | 397 +++++++++++++++++++++++
 ofproto/ofproto-dpif-recirculation.h |  81 +++++
 ofproto/ofproto-dpif-upcall.c        |  33 +-
 ofproto/ofproto-dpif-xlate.c         | 247 +++++++++++++-
 ofproto/ofproto-dpif-xlate.h         |  27 +-
 ofproto/ofproto-dpif.c               |  22 +-
 ofproto/ofproto-dpif.h               |   6 +-
 ofproto/tunnel.c                     |   3 -
 ofproto/tunnel.h                     |   3 +
 tests/ofproto-dpif.at                | 613 +++++++++++++++++++++++++++++++++++
 utilities/ovs-ofctl.8.in             |   5 -
 26 files changed, 1564 insertions(+), 114 deletions(-)
 create mode 100644 ofproto/ofproto-dpif-recirculation.c
 create mode 100644 ofproto/ofproto-dpif-recirculation.h

diff --git a/datapath/actions.c b/datapath/actions.c
index 30ea1d2..88c3cd2 100644
--- a/datapath/actions.c
+++ b/datapath/actions.c
@@ -553,6 +553,9 @@ static int do_execute_actions(struct datapath *dp, struct sk_buff *skb,
 		case OVS_ACTION_ATTR_SAMPLE:
 			err = sample(dp, skb, a);
 			break;
+
+		case OVS_ACTION_ATTR_RECIRCULATE:
+			return 1;
 		}
 
 		if (unlikely(err)) {
@@ -593,7 +596,7 @@ static int loop_suppress(struct datapath *dp, struct sw_flow_actions *actions)
 }
 
 /* Execute a list of actions against 'skb'. */
-int ovs_execute_actions(struct datapath *dp, struct sk_buff *skb)
+struct sk_buff *ovs_execute_actions(struct datapath *dp, struct sk_buff *skb)
 {
 	struct sw_flow_actions *acts = rcu_dereference(OVS_CB(skb)->flow->sf_acts);
 	struct loop_counter *loop;
@@ -612,6 +615,8 @@ int ovs_execute_actions(struct datapath *dp, struct sk_buff *skb)
 	OVS_CB(skb)->tun_key = NULL;
 	error = do_execute_actions(dp, skb, acts->actions,
 					 acts->actions_len, false);
+	if (likely(error <= 0))
+		skb = NULL;
 
 	/* Check whether sub-actions looped too much. */
 	if (unlikely(loop->looping))
@@ -622,5 +627,5 @@ out_loop:
 	if (!--loop->count)
 		loop->looping = false;
 
-	return error;
+	return (error < 0) ? ERR_PTR(error) : skb;
 }
diff --git a/datapath/datapath.c b/datapath/datapath.c
index d528ba0..77b08d4 100644
--- a/datapath/datapath.c
+++ b/datapath/datapath.c
@@ -214,54 +214,67 @@ void ovs_dp_detach_port(struct vport *p)
 	ovs_vport_del(p);
 }
 
+#define MAX_RECIRCULATION_DEPTH	4	/* Completely arbitrary */
+
 /* Must be called with rcu_read_lock. */
 void ovs_dp_process_received_packet(struct vport *p, struct sk_buff *skb)
 {
 	struct datapath *dp = p->dp;
-	struct sw_flow *flow;
 	struct dp_stats_percpu *stats;
-	struct sw_flow_key key;
-	u64 *stats_counter;
-	u32 n_mask_hit;
-	int error;
+	int limit = MAX_RECIRCULATION_DEPTH;
 
 	stats = this_cpu_ptr(dp->stats_percpu);
 
-	/* Extract flow from 'skb' into 'key'. */
-	error = ovs_flow_extract(skb, p->port_no, &key);
-	if (unlikely(error)) {
-		kfree_skb(skb);
-		return;
-	}
+	while (1) {
+		u64 *stats_counter;
+		struct sw_flow *flow;
+		struct sw_flow_key key;
+		u32 n_mask_hit;
+		int error;
 
-	/* Look up flow. */
-	flow = ovs_flow_tbl_lookup_stats(&dp->table, &key, &n_mask_hit);
-	if (unlikely(!flow)) {
-		struct dp_upcall_info upcall;
-
-		upcall.cmd = OVS_PACKET_CMD_MISS;
-		upcall.key = &key;
-		upcall.userdata = NULL;
-		upcall.portid = p->upcall_portid;
-		ovs_dp_upcall(dp, skb, &upcall);
-		consume_skb(skb);
-		stats_counter = &stats->n_missed;
-		goto out;
-	}
+		/* Extract flow from 'skb' into 'key'. */
+		error = ovs_flow_extract(skb, p->port_no, &key);
+		if (unlikely(error)) {
+			kfree_skb(skb);
+			return;
+		}
+
+		/* Look up flow. */
+		flow = ovs_flow_tbl_lookup_stats(&dp->table, &key, &n_mask_hit);
+		if (unlikely(!flow)) {
+			struct dp_upcall_info upcall;
 
-	OVS_CB(skb)->flow = flow;
-	OVS_CB(skb)->pkt_key = &key;
+			upcall.cmd = OVS_PACKET_CMD_MISS;
+			upcall.key = &key;
+			upcall.userdata = NULL;
+			upcall.portid = p->upcall_portid;
+			ovs_dp_upcall(dp, skb, &upcall);
+			consume_skb(skb);
+			stats_counter = &stats->n_missed;
+		} else {
+			OVS_CB(skb)->flow = flow;
+			OVS_CB(skb)->pkt_key = &key;
+			ovs_flow_stats_update(OVS_CB(skb)->flow, skb);
+			ovs_execute_actions(dp, skb);
+			stats_counter = &stats->n_hit;
+		}
 
-	ovs_flow_stats_update(OVS_CB(skb)->flow, skb);
-	ovs_execute_actions(dp, skb);
-	stats_counter = &stats->n_hit;
+		/* Update datapath statistics. */
+		u64_stats_update_begin(&stats->sync);
+		(*stats_counter)++;
+		stats->n_mask_hit += n_mask_hit;
+		u64_stats_update_end(&stats->sync);
 
-out:
-	/* Update datapath statistics. */
-	u64_stats_update_begin(&stats->sync);
-	(*stats_counter)++;
-	stats->n_mask_hit += n_mask_hit;
-	u64_stats_update_end(&stats->sync);
+
+		if (IS_ERR_OR_NULL(skb)) {
+			return;
+		} else if (unlikely(!--limit)) {
+			net_warn_ratelimited("droped packet that has "
+					     "been recirculated too much");
+			kfree_skb(skb);
+			return;
+		}
+	}
 }
 
 static struct genl_family dp_packet_genl_family = {
@@ -560,12 +573,23 @@ static int ovs_packet_cmd_execute(struct sk_buff *skb, struct genl_info *info)
 		goto err_unlock;
 
 	local_bh_disable();
-	err = ovs_execute_actions(dp, packet);
+	packet = ovs_execute_actions(dp, packet);
+	if (!IS_ERR_OR_NULL(packet)) {
+		struct vport *vport;
+		vport = ovs_lookup_vport(dp, flow->key.phy.in_port);
+		if (!vport) {
+			err = -ENODEV;
+			goto err_unlock;
+		}
+		/* Recirculate */
+		ovs_dp_process_received_packet(vport, packet);
+		packet = NULL;
+	}
 	local_bh_enable();
 	rcu_read_unlock();
 
 	ovs_flow_free(flow, false);
-	return err;
+	return PTR_ERR(packet);
 
 err_unlock:
 	rcu_read_unlock();
diff --git a/datapath/datapath.h b/datapath/datapath.h
index d81e05c..31bb73c 100644
--- a/datapath/datapath.h
+++ b/datapath/datapath.h
@@ -195,7 +195,7 @@ const char *ovs_dp_name(const struct datapath *dp);
 struct sk_buff *ovs_vport_cmd_build_info(struct vport *, u32 portid, u32 seq,
 					 u8 cmd);
 
-int ovs_execute_actions(struct datapath *dp, struct sk_buff *skb);
+struct sk_buff *ovs_execute_actions(struct datapath *dp, struct sk_buff *skb);
 void ovs_dp_notify_wq(struct work_struct *work);
 
 #define OVS_NLERR(fmt, ...)					\
diff --git a/datapath/flow_netlink.c b/datapath/flow_netlink.c
index 39fe4bf..535a395 100644
--- a/datapath/flow_netlink.c
+++ b/datapath/flow_netlink.c
@@ -1507,6 +1507,7 @@ int ovs_nla_copy_actions(const struct nlattr *attr,
 {
 	const struct nlattr *a;
 	int rem, err;
+	bool recirculate = false;
 
 	if (depth >= SAMPLE_ACTION_DEPTH)
 		return -EOVERFLOW;
@@ -1518,6 +1519,7 @@ int ovs_nla_copy_actions(const struct nlattr *attr,
 			[OVS_ACTION_ATTR_USERSPACE] = (u32)-1,
 			[OVS_ACTION_ATTR_PUSH_VLAN] = sizeof(struct ovs_action_push_vlan),
 			[OVS_ACTION_ATTR_POP_VLAN] = 0,
+			[OVS_ACTION_ATTR_RECIRCULATE] = 0,
 			[OVS_ACTION_ATTR_SET] = (u32)-1,
 			[OVS_ACTION_ATTR_SAMPLE] = (u32)-1
 		};
@@ -1525,6 +1527,9 @@ int ovs_nla_copy_actions(const struct nlattr *attr,
 		int type = nla_type(a);
 		bool skip_copy;
 
+		if (recirculate)
+			break;
+
 		if (type > OVS_ACTION_ATTR_MAX ||
 		    (action_lens[type] != nla_len(a) &&
 		     action_lens[type] != (u32)-1))
@@ -1571,6 +1576,10 @@ int ovs_nla_copy_actions(const struct nlattr *attr,
 			skip_copy = true;
 			break;
 
+		case OVS_ACTION_ATTR_RECIRCULATE:
+			recirculate = true;
+			break;
+
 		default:
 			return -EINVAL;
 		}
diff --git a/include/linux/openvswitch.h b/include/linux/openvswitch.h
index d1ff5ec..c21506d 100644
--- a/include/linux/openvswitch.h
+++ b/include/linux/openvswitch.h
@@ -553,6 +553,9 @@ struct ovs_action_push_vlan {
  * indicate the new packet contents. This could potentially still be
  * %ETH_P_MPLS if the resulting MPLS label stack is not empty.  If there
  * is no MPLS label stack, as determined by ethertype, no action is taken.
+ * @OVS_ACTION_ATTR_RECIRCULATE: Restart processing of packet.
+ * The packet must have been modified by a previous action in such a way
+ * that it does not match its original flow again.
  *
  * Only a single header can be set with a single %OVS_ACTION_ATTR_SET.  Not all
  * fields within a header are modifiable, e.g. the IPv4 protocol and fragment
@@ -569,6 +572,7 @@ enum ovs_action_attr {
 	OVS_ACTION_ATTR_SAMPLE,       /* Nested OVS_SAMPLE_ATTR_*. */
 	OVS_ACTION_ATTR_PUSH_MPLS,    /* struct ovs_action_push_mpls. */
 	OVS_ACTION_ATTR_POP_MPLS,     /* __be16 ethertype. */
+	OVS_ACTION_ATTR_RECIRCULATE,  /* No argument */
 	__OVS_ACTION_ATTR_MAX
 };
 
diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 73eb99d..9f03c30 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -323,7 +323,7 @@ static int dp_netdev_output_userspace(struct dp_netdev *dp, struct ofpbuf *,
                                     int queue_no, const struct flow *,
                                     const struct nlattr *userdata)
     OVS_EXCLUDED(dp->queue_mutex);
-static void dp_netdev_execute_actions(struct dp_netdev *dp,
+static bool dp_netdev_execute_actions(struct dp_netdev *dp,
                                       const struct flow *, struct ofpbuf *,
                                       struct pkt_metadata *,
                                       const struct nlattr *actions,
@@ -1410,6 +1410,8 @@ dpif_netdev_execute(struct dpif *dpif, struct dpif_execute *execute)
     struct dp_netdev *dp = get_dp_netdev(dpif);
     struct pkt_metadata *md = &execute->md;
     struct flow key;
+    bool recirculate;
+    int error = 0;
 
     if (execute->packet->size < ETH_HEADER_LEN ||
         execute->packet->size > UINT16_MAX) {
@@ -1421,11 +1423,15 @@ dpif_netdev_execute(struct dpif *dpif, struct dpif_execute *execute)
                  (union flow_in_port *)&md->in_port, &key);
 
     ovs_rwlock_rdlock(&dp->port_rwlock);
-    dp_netdev_execute_actions(dp, &key, execute->packet, md, execute->actions,
-                              execute->actions_len);
+    recirculate = dp_netdev_execute_actions(dp, &key, execute->packet, md,
+                                            execute->actions,
+                                            execute->actions_len);
+    if (recirculate) {
+        dp_netdev_port_input(dp, execute->packet, md);
+    }
     ovs_rwlock_unlock(&dp->port_rwlock);
 
-    return 0;
+    return error;
 }
 
 static int
@@ -1679,16 +1685,17 @@ dp_netdev_flow_used(struct dp_netdev_flow *netdev_flow,
     netdev_flow->tcp_flags |= packet_get_tcp_flags(packet, &netdev_flow->flow);
 }
 
-static void
-dp_netdev_port_input(struct dp_netdev *dp, struct ofpbuf *packet,
-                     struct pkt_metadata *md)
+static bool
+__dp_netdev_port_input(struct dp_netdev *dp, struct ofpbuf *packet,
+                       struct pkt_metadata *md)
     OVS_REQ_RDLOCK(dp->port_rwlock)
 {
     struct dp_netdev_flow *netdev_flow;
+    bool recirculate = false;
     struct flow key;
 
     if (packet->size < ETH_HEADER_LEN) {
-        return;
+        return false;
     }
     flow_extract(packet, md->skb_priority, md->pkt_mark, &md->tunnel,
                  (union flow_in_port *)&md->in_port, &key);
@@ -1701,14 +1708,30 @@ dp_netdev_port_input(struct dp_netdev *dp, struct ofpbuf *packet,
         actions = dp_netdev_actions_ref(netdev_flow->actions);
         ovs_mutex_unlock(&netdev_flow->mutex);
 
-        dp_netdev_execute_actions(dp, &key, packet, md,
-                                  actions->actions, actions->size);
+        recirculate = dp_netdev_execute_actions(dp, &key, packet, md,
+                                                actions->actions,
+                                                actions->size);
         dp_netdev_actions_unref(actions);
         ovsthread_counter_inc(dp->n_hit, 1);
     } else {
         ovsthread_counter_inc(dp->n_missed, 1);
         dp_netdev_output_userspace(dp, packet, DPIF_UC_MISS, &key, NULL);
     }
+
+    return recirculate;
+}
+
+static void
+dp_netdev_port_input(struct dp_netdev *dp, struct ofpbuf *packet,
+                     struct pkt_metadata *md)
+    OVS_REQ_RDLOCK(dp->port_rwlock)
+{
+    bool recirculate;
+    int limit = MAX_RECIRCULATION_DEPTH;
+
+    do {
+        recirculate = __dp_netdev_port_input(dp, packet, md);
+    } while(recirculate && --limit);
 }
 
 static int
@@ -1810,12 +1833,13 @@ dp_execute_cb(void *aux_, struct ofpbuf *packet,
     case OVS_ACTION_ATTR_SET:
     case OVS_ACTION_ATTR_SAMPLE:
     case OVS_ACTION_ATTR_UNSPEC:
+    case OVS_ACTION_ATTR_RECIRCULATE:
     case __OVS_ACTION_ATTR_MAX:
         OVS_NOT_REACHED();
     }
 }
 
-static void
+static bool
 dp_netdev_execute_actions(struct dp_netdev *dp, const struct flow *key,
                           struct ofpbuf *packet, struct pkt_metadata *md,
                           const struct nlattr *actions, size_t actions_len)
@@ -1823,7 +1847,8 @@ dp_netdev_execute_actions(struct dp_netdev *dp, const struct flow *key,
 {
     struct dp_netdev_execute_aux aux = {dp, key};
 
-    odp_execute_actions(&aux, packet, md, actions, actions_len, dp_execute_cb);
+    return odp_execute_actions(&aux, packet, md, actions, actions_len,
+                               dp_execute_cb);
 }
 
 const struct dpif_class dpif_netdev_class = {
diff --git a/lib/dpif.c b/lib/dpif.c
index 2b79a6e..4c5ca40 100644
--- a/lib/dpif.c
+++ b/lib/dpif.c
@@ -1093,6 +1093,7 @@ dpif_execute_helper_cb(void *aux_, struct ofpbuf *packet,
     case OVS_ACTION_ATTR_SET:
     case OVS_ACTION_ATTR_SAMPLE:
     case OVS_ACTION_ATTR_UNSPEC:
+    case OVS_ACTION_ATTR_RECIRCULATE:
     case __OVS_ACTION_ATTR_MAX:
         OVS_NOT_REACHED();
     }
diff --git a/lib/flow.h b/lib/flow.h
index 498502e..448295c 100644
--- a/lib/flow.h
+++ b/lib/flow.h
@@ -470,4 +470,7 @@ minimask_get_metadata_mask(const struct minimask *mask)
     return miniflow_get_metadata(&mask->masks);
 }
 
+#define MAX_RECIRCULATION_DEPTH 4   /* Completely arbitrary value to
+                                     * guard against infinite loops */
+BUILD_ASSERT_DECL(MAX_RECIRCULATION_DEPTH > 0);
 #endif /* flow.h */
diff --git a/lib/odp-execute.c b/lib/odp-execute.c
index 096c113..ec53b4c 100644
--- a/lib/odp-execute.c
+++ b/lib/odp-execute.c
@@ -140,12 +140,12 @@ odp_execute_set_action(struct ofpbuf *packet, const struct nlattr *a,
     }
 }
 
-static void
+static bool
 odp_execute_actions__(void *dp, struct ofpbuf *packet, struct pkt_metadata *,
                       const struct nlattr *actions, size_t actions_len,
                       odp_execute_cb dp_execute_action, bool more_actions);
 
-static void
+static bool
 odp_execute_sample(void *dp, struct ofpbuf *packet, struct pkt_metadata *md,
                    const struct nlattr *action,
                    odp_execute_cb dp_execute_action, bool more_actions)
@@ -160,7 +160,7 @@ odp_execute_sample(void *dp, struct ofpbuf *packet, struct pkt_metadata *md,
         switch ((enum ovs_sample_attr) type) {
         case OVS_SAMPLE_ATTR_PROBABILITY:
             if (random_uint32() >= nl_attr_get_u32(a)) {
-                return;
+                return false;
             }
             break;
 
@@ -175,12 +175,12 @@ odp_execute_sample(void *dp, struct ofpbuf *packet, struct pkt_metadata *md,
         }
     }
 
-    odp_execute_actions__(dp, packet, md, nl_attr_get(subactions),
-                          nl_attr_get_size(subactions), dp_execute_action,
-                          more_actions);
+    return odp_execute_actions__(dp, packet, md, nl_attr_get(subactions),
+                                 nl_attr_get_size(subactions),
+                                 dp_execute_action, more_actions);
 }
 
-static void
+static bool
 odp_execute_actions__(void *dp, struct ofpbuf *packet, struct pkt_metadata *md,
                       const struct nlattr *actions, size_t actions_len,
                       odp_execute_cb dp_execute_action, bool more_actions)
@@ -228,22 +228,30 @@ odp_execute_actions__(void *dp, struct ofpbuf *packet, struct pkt_metadata *md,
             break;
 
         case OVS_ACTION_ATTR_SAMPLE:
-            odp_execute_sample(dp, packet, md, a, dp_execute_action,
-                               more_actions || left > NLA_ALIGN(a->nla_len));
+            if (odp_execute_sample(dp, packet, md, a, dp_execute_action,
+                                   more_actions || left >
+                                   NLA_ALIGN(a->nla_len))) {
+                return true;
+            }
             break;
 
+        case OVS_ACTION_ATTR_RECIRCULATE:
+            return true;
+
         case OVS_ACTION_ATTR_UNSPEC:
         case __OVS_ACTION_ATTR_MAX:
             OVS_NOT_REACHED();
         }
     }
+
+    return false;
 }
 
-void
+bool
 odp_execute_actions(void *dp, struct ofpbuf *packet, struct pkt_metadata *md,
                     const struct nlattr *actions, size_t actions_len,
                     odp_execute_cb dp_execute_action)
 {
-    odp_execute_actions__(dp, packet, md, actions, actions_len,
-                          dp_execute_action, false);
+    return odp_execute_actions__(dp, packet, md, actions, actions_len,
+                                 dp_execute_action, false);
 }
diff --git a/lib/odp-execute.h b/lib/odp-execute.h
index 670e8ea..ba33870 100644
--- a/lib/odp-execute.h
+++ b/lib/odp-execute.h
@@ -35,7 +35,7 @@ typedef void (*odp_execute_cb)(void *dp, struct ofpbuf *packet,
  * to 'dp_execute_action', if non-NULL.  Currently this is called only for
  * actions OVS_ACTION_ATTR_OUTPUT and OVS_ACTION_ATTR_USERSPACE so
  * 'dp_execute_action' needs to handle only these. */
-void
+bool
 odp_execute_actions(void *dp, struct ofpbuf *packet, struct pkt_metadata *,
                     const struct nlattr *actions, size_t actions_len,
                     odp_execute_cb dp_execute_action);
diff --git a/lib/odp-util.c b/lib/odp-util.c
index e20564f..8316304 100644
--- a/lib/odp-util.c
+++ b/lib/odp-util.c
@@ -79,6 +79,7 @@ odp_action_len(uint16_t type)
     case OVS_ACTION_ATTR_POP_VLAN: return 0;
     case OVS_ACTION_ATTR_PUSH_MPLS: return sizeof(struct ovs_action_push_mpls);
     case OVS_ACTION_ATTR_POP_MPLS: return sizeof(ovs_be16);
+    case OVS_ACTION_ATTR_RECIRCULATE: return 0;
     case OVS_ACTION_ATTR_SET: return -2;
     case OVS_ACTION_ATTR_SAMPLE: return -2;
 
@@ -435,6 +436,10 @@ format_odp_action(struct ds *ds, const struct nlattr *a)
         ds_put_format(ds, "pop_mpls(eth_type=0x%"PRIx16")", ntohs(ethertype));
         break;
     }
+    case OVS_ACTION_ATTR_RECIRCULATE: {
+        ds_put_format(ds, "recirculate");
+        break;
+    }
     case OVS_ACTION_ATTR_SAMPLE:
         format_odp_sample_action(ds, a);
         break;
@@ -3436,6 +3441,12 @@ commit_odp_tunnel_action(const struct flow *flow, struct flow *base,
     }
 }
 
+void
+commit_odp_recirculate_action(struct ofpbuf *odp_actions)
+{
+    nl_msg_put_flag(odp_actions, OVS_ACTION_ATTR_RECIRCULATE);
+}
+
 static void
 commit_set_ether_addr_action(const struct flow *flow, struct flow *base,
                              struct ofpbuf *odp_actions,
@@ -3770,8 +3781,8 @@ commit_set_pkt_mark_action(const struct flow *flow, struct flow *base,
  * 'base' and 'flow', appends ODP actions to 'odp_actions' that change the flow
  * key from 'base' into 'flow', and then changes 'base' the same way.  Does not
  * commit set_tunnel actions.  Users should call commit_odp_tunnel_action()
- * in addition to this function if needed.  Sets fields in 'wc' that are
- * used as part of the action.
+ * and commit_odp_recirculate_action() in addition to those functions if
+ * needed. Sets fields in 'wc' that are used as part of the action.
  *
  * Returns a reason to force processing the flow's packets into the userspace
  * slow path, if there is one, otherwise 0. */
diff --git a/lib/odp-util.h b/lib/odp-util.h
index 7bc64c7..2977fff 100644
--- a/lib/odp-util.h
+++ b/lib/odp-util.h
@@ -179,6 +179,7 @@ const char *odp_key_fitness_to_string(enum odp_key_fitness);
 
 void commit_odp_tunnel_action(const struct flow *, struct flow *base,
                               struct ofpbuf *odp_actions);
+void commit_odp_recirculate_action(struct ofpbuf *odp_actions);
 enum slow_path_reason commit_odp_actions(const struct flow *,
                                          struct flow *base,
                                          struct ofpbuf *odp_actions,
diff --git a/lib/ofp-actions.c b/lib/ofp-actions.c
index 781c3a1..6461a36 100644
--- a/lib/ofp-actions.c
+++ b/lib/ofp-actions.c
@@ -491,9 +491,6 @@ ofpact_from_nxast(const union ofp_action *a, enum ofputil_action_code code,
         break;
 
     case OFPUTIL_NXAST_POP_MPLS:
-        if (eth_type_mpls(a->pop_mpls.ethertype)) {
-            return OFPERR_OFPBAC_BAD_ARGUMENT;
-        }
         ofpact_put_POP_MPLS(out)->ethertype = a->pop_mpls.ethertype;
         break;
 
@@ -1256,9 +1253,6 @@ ofpact_from_openflow11(const union ofp_action *a, enum ofp_version version,
         break;
 
     case OFPUTIL_OFPAT11_POP_MPLS:
-        if (eth_type_mpls(a->ofp11_pop_mpls.ethertype)) {
-            return OFPERR_OFPBAC_BAD_ARGUMENT;
-        }
         ofpact_put_POP_MPLS(out)->ethertype = a->ofp11_pop_mpls.ethertype;
         break;
 
diff --git a/lib/ofp-actions.h b/lib/ofp-actions.h
index 0f6bf70..5b843d2 100644
--- a/lib/ofp-actions.h
+++ b/lib/ofp-actions.h
@@ -182,6 +182,12 @@ ofpact_end(const struct ofpact *ofpacts, size_t ofpacts_len)
     return (void *) ((uint8_t *) ofpacts + ofpacts_len);
 }
 
+static inline unsigned
+ofpact_offset(const struct ofpact *start, const struct ofpact *offset)
+{
+    return (uint8_t*) offset - (uint8_t *) start;
+}
+
 /* Assigns POS to each ofpact, in turn, in the OFPACTS_LEN bytes of ofpacts
  * starting at OFPACTS. */
 #define OFPACT_FOR_EACH(POS, OFPACTS, OFPACTS_LEN)                      \
diff --git a/ofproto/automake.mk b/ofproto/automake.mk
index 1308820..9151b33 100644
--- a/ofproto/automake.mk
+++ b/ofproto/automake.mk
@@ -31,6 +31,8 @@ ofproto_libofproto_la_SOURCES = \
 	ofproto/ofproto-dpif-mirror.h \
 	ofproto/ofproto-dpif-monitor.c \
 	ofproto/ofproto-dpif-monitor.h \
+	ofproto/ofproto-dpif-recirculation.c \
+	ofproto/ofproto-dpif-recirculation.h \
 	ofproto/ofproto-dpif-sflow.c \
 	ofproto/ofproto-dpif-sflow.h \
 	ofproto/ofproto-dpif-upcall.c \
diff --git a/ofproto/ofproto-dpif-recirculation.c b/ofproto/ofproto-dpif-recirculation.c
new file mode 100644
index 0000000..8c0ac9d
--- /dev/null
+++ b/ofproto/ofproto-dpif-recirculation.c
@@ -0,0 +1,397 @@
+/*
+ * Copyright (c) 2013 Nicira, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at:
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <config.h>
+
+#include "dynamic-string.h"
+#include "ofp-actions.h"
+#include "ofproto-dpif-recirculation.h"
+#include "ofproto-dpif-xlate.h"
+#include "ofproto-provider.h"
+
+#include "hash.h"
+#include "vlog.h"
+
+VLOG_DEFINE_THIS_MODULE(ofproto_dpif_recirculation);
+
+#define RECIRCULATION_ID_MAX_LOOP 1024  /* Arbitrary value to prevent
+                                         * endless loop */
+
+/* XXX: Associate with backer rather than being global? */
+static struct ovs_rwlock recirculator_rwlock = OVS_RWLOCK_INITIALIZER;
+static struct hmap recirculator_id_hmap OVS_GUARDED_BY(recirculator_rwlock) =
+    HMAP_INITIALIZER(&recirculator_id_hmap);
+static struct hmap recirculator_flow_hmap OVS_GUARDED_BY(recirculator_rwlock) =
+    HMAP_INITIALIZER(&recirculator_flow_hmap);
+
+static uint32_t recirculation_id_hash(uint32_t id)
+{
+    return hash_words(&id, 1, 0);
+}
+
+static void
+recirculator_destroy(struct recirculator *r)
+    OVS_EXCLUDED(recirculator_rwlock)
+{
+    ovs_rwlock_wrlock(&recirculator_rwlock);
+    hmap_remove(&recirculator_id_hmap, &r->id_hmap_node);
+    hmap_remove(&recirculator_flow_hmap, &r->flow_hmap_node);
+    ofpbuf_uninit(r->odp_actions);
+    free(r);
+    ovs_rwlock_unlock(&recirculator_rwlock);
+}
+
+static void
+recirculator_create(uint32_t id, const struct flow *flow,
+                    size_t ofpacts_offset, size_t ofpacts_len,
+                    const struct ofpbuf *odp_actions)
+    OVS_REQ_WRLOCK(recirculator_rwlock)
+{
+    uint32_t hash;
+    struct recirculator *r;
+
+    r = xmalloc(sizeof *r);
+
+    atomic_init(&r->ref_cnt, 1);
+    r->id = id;
+    r->flow = *flow;
+    r->ofpacts_offset = ofpacts_offset;
+    r->ofpacts_len = ofpacts_len;
+    r->odp_actions = ofpbuf_clone(odp_actions);
+
+    hash = recirculation_id_hash(id);
+    hmap_insert(&recirculator_id_hmap, &r->id_hmap_node, hash);
+
+    hash = flow_hash(flow, 0);
+    hmap_insert(&recirculator_flow_hmap, &r->flow_hmap_node, hash);
+}
+
+void
+recirculator_unref(struct recirculator *r)
+    OVS_EXCLUDED(recirculator_rwlock)
+{
+    int orig;
+
+    if (!r) {
+        return;
+    }
+
+    atomic_sub(&r->ref_cnt, 1, &orig);
+    ovs_assert(orig > 0);
+    if (orig == 1) {
+        recirculator_destroy(r);
+    }
+}
+
+static inline void
+recirculator_ref(struct recirculator *r)
+{
+    int orig;
+
+    if (!r) {
+        return;
+    }
+
+    atomic_add(&r->ref_cnt, 1, &orig);
+    ovs_assert(orig > 0);
+}
+
+static struct recirculator *
+recirculator_lookup_by_flow_and_odp_actions_nolock_noref(const struct flow *flow,
+                                                         const struct ofpbuf *odp_actions)
+    OVS_REQ_RDLOCK(recirculator_rwlock)
+{
+    struct recirculator *r;
+    uint32_t hash;
+
+    hash = flow_hash(flow, 0);
+    HMAP_FOR_EACH_WITH_HASH (r, flow_hmap_node, hash, &recirculator_flow_hmap) {
+        if (flow_equal(flow, &r->flow) &&
+            ofpbuf_equal(r->odp_actions, odp_actions)) {
+            return r;
+        }
+    }
+
+    return NULL;
+}
+
+static struct recirculator *
+recirculator_lookup_by_id_nolock_noref(uint32_t id)
+    OVS_REQ_RDLOCK(recirculator_rwlock)
+{
+    struct recirculator *r;
+    uint32_t hash;
+
+    hash = recirculation_id_hash(id);
+    HMAP_FOR_EACH_WITH_HASH (r, id_hmap_node, hash, &recirculator_id_hmap) {
+        if (r->id == id) {
+            return r;
+        }
+    }
+
+    return NULL;
+}
+
+static struct recirculator *
+recirculator_lookup_by_id_nolock(uint32_t id)
+    OVS_REQ_RDLOCK(recirculator_rwlock)
+{
+    struct recirculator *r;
+
+    r = recirculator_lookup_by_id_nolock_noref(id);
+    recirculator_ref(r);
+
+    return r;
+}
+
+static struct recirculator *
+recirculator_lookup_by_pkt_mark_noref(uint32_t pkt_mark)
+    OVS_EXCLUDED(recirculator_rwlock)
+{
+    struct recirculator *r;
+    uint32_t id;
+
+    id = recirculation_id_from_pkt_mark(pkt_mark);
+
+    if (!recirculation_id_valid(id)) {
+        return false;
+    }
+
+    ovs_rwlock_rdlock(&recirculator_rwlock);
+    r = recirculator_lookup_by_id_nolock_noref(id);
+    ovs_rwlock_unlock(&recirculator_rwlock);
+
+    return r;
+}
+
+bool
+recirculator_exists(uint32_t pkt_mark)
+    OVS_EXCLUDED(recirculator_rwlock)
+{
+    return recirculator_lookup_by_pkt_mark_noref(pkt_mark) != NULL;
+}
+
+void
+recirculator_delete(uint32_t pkt_mark)
+    OVS_EXCLUDED(recirculator_rwlock)
+{
+    struct recirculator *r;
+
+    r = recirculator_lookup_by_pkt_mark_noref(pkt_mark);
+    recirculator_unref(r);
+}
+
+uint32_t
+recirculation_id_get(const struct flow *flow, size_t ofpacts_offset,
+                     size_t ofpacts_len, const struct ofpbuf *odp_actions)
+    OVS_EXCLUDED(recirculator_rwlock)
+{
+    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 15);
+    static uint32_t id OVS_GUARDED_BY(recirculator_rwlock) =
+        RECIRCULATION_ID_MIN;
+
+    struct recirculator *r;
+    int loop = RECIRCULATION_ID_MAX_LOOP;
+    uint32_t ret = RECIRCULATION_ID_DUMMY;
+
+    ovs_rwlock_wrlock(&recirculator_rwlock);
+    r = recirculator_lookup_by_flow_and_odp_actions_nolock_noref(flow,
+                                                                 odp_actions);
+    if (r) {
+        ret = r->id;
+        goto unlock;
+    }
+
+    while (loop--) {
+        if (recirculation_id_valid(id)) {
+            r = recirculator_lookup_by_id_nolock(id);
+            if (!r) {
+                recirculator_create(id, flow, ofpacts_offset,
+                                    ofpacts_len, odp_actions);
+                ret = id;
+                break;
+            }
+        }
+        id++;
+    }
+
+unlock:
+    ovs_rwlock_unlock(&recirculator_rwlock);
+
+    if (ret == RECIRCULATION_ID_DUMMY) {
+        VLOG_WARN_RL(&rl,
+                     "Failed to allocate recirulation id after %d attempts\n",
+                     RECIRCULATION_ID_MAX_LOOP);
+    }
+
+    return ret;
+}
+
+/* The length ddof the ODP actions added on recirculation */
+static size_t
+reciculation_odp_actions_len(void)
+{
+    static size_t len = 0;
+
+    if (len == 0) {
+        struct ofpbuf odp_actions;
+        ofpbuf_init(&odp_actions, 32);
+        odp_put_pkt_mark_action(0, &odp_actions);
+        commit_odp_recirculate_action(&odp_actions);
+        len = odp_actions.size;
+        ofpbuf_uninit(&odp_actions);
+    }
+
+    return len;
+}
+
+bool
+recirculator_chain_verify(struct ofproto_dpif *ofproto,
+                          struct rule_dpif *rule, uint8_t tcp_flags,
+                          struct recirculator **chain, size_t len)
+{
+    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 15);
+    struct recirculator *prev_r = NULL;
+    size_t i;
+
+    for (i = len; i > 0; i--) {
+        struct xlate_in xin;
+        struct recirculator *r = chain[i - 1];
+        struct xlate_out xout;
+
+        xlate_in_init(&xin, ofproto, &r->flow, rule, tcp_flags, NULL, r->id);
+        if (prev_r) {
+            xlate_in_init_ofpacts(&xin, prev_r->ofpacts_offset, prev_r->ofpacts_len);
+        }
+
+        xlate_actions(&xin, &xout);
+
+        if (!xin.recirculated && i > 1) {
+            VLOG_WARN_RL(&rl, "Recirculation chain verification failed: "
+                         "not recirculated depth depth %"PRIuSIZE" of "
+                         "%"PRIuSIZE".", i, len);
+            xlate_out_uninit(&xout);
+            return false;
+        }
+
+        if (xin.ofpacts_len != r->ofpacts_len) {
+            VLOG_WARN_RL(&rl, "Recirculation chain verification failed: "
+                         "flow ofpacts miss-match at depth %"PRIuSIZE" of "
+                         "%"PRIuSIZE". Recirculator length %"PRIuSIZE". "
+                         "Translation yielded length %"PRIuSIZE".", i, len,
+                         r->ofpacts_len, xin.ofpacts_len);
+            xlate_out_uninit(&xout);
+            return false;
+        }
+
+        /* r->odp_actions->size should be reciculation_odp_actions_len()
+         * shorter than xout.odp_actions.size as it is missing the final
+         * set pkt_mark and recirculate actions */
+        if (r->odp_actions->size + reciculation_odp_actions_len() !=
+            xout.odp_actions.size &&
+            memcmp(r->odp_actions->data, xout.odp_actions.data,
+                   r->odp_actions->size) == 0) {
+            struct ds ds = DS_EMPTY_INITIALIZER;
+            char *s;
+
+            ds_put_format(&ds, "Recirculation chain verification failed: "
+                          "actions miss-match in recirculation chain "
+                          "at depth %"PRIuSIZE" of %"PRIuSIZE". "
+                          "Recirculator has: ", i, len);
+            format_odp_actions(&ds, r->odp_actions->data, r->odp_actions->size);
+
+            ds_put_cstr(&ds, ". Translation yielded: ");
+            format_odp_actions(&ds, xout.odp_actions.data,
+                               xout.odp_actions.size);
+
+            ds_put_char(&ds, '.');
+
+            s = ds_steal_cstr(&ds);
+            VLOG_WARN_RL(&rl, "%s", s);
+            free(s);
+
+            xlate_out_uninit(&xout);
+            return false;
+        }
+
+        xlate_out_uninit(&xout);
+        prev_r = r;
+    }
+
+    return true;
+}
+
+size_t
+recirculator_chain_lookup(uint32_t pkt_mark, struct recirculator **chain,
+                          size_t len)
+    OVS_EXCLUDED(recirculator_rwlock)
+{
+    static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 15);
+    size_t top = 0;
+    uint32_t id;
+
+    ovs_assert(len > 0);
+
+    id = recirculation_id_from_pkt_mark(pkt_mark);
+    if (!recirculation_id_valid(id)) {
+        return 0;
+    }
+
+    ovs_rwlock_rdlock(&recirculator_rwlock);
+    while (recirculation_id_valid(id) && top < len) {
+        chain[top] = recirculator_lookup_by_id_nolock(id);
+        if (!chain[top]) {
+            ovs_rwlock_unlock(&recirculator_rwlock);
+            goto unref;
+        }
+        id = chain[top]->flow.pkt_mark;
+        top++;
+    }
+    ovs_rwlock_unlock(&recirculator_rwlock);
+
+    if (recirculation_id_valid(chain[top - 1]->flow.pkt_mark)) {
+        VLOG_WARN_RL(&rl, "root recirculation id for recirculation id %d "
+                    "could not be found due to under-size buffer",
+                    chain[0]->id);
+        goto unref;
+    }
+
+    return top;
+
+unref:
+    recirculator_chain_unref(chain, top);
+    return 0;
+}
+
+void
+recirculator_chain_unref(struct recirculator **chain, size_t len)
+{
+    while (len--) {
+        if (chain[len]) {
+             recirculator_unref(chain[len]);
+        }
+    }
+}
+
+void
+recirculate_xlate_init(struct flow *flow, struct flow_wildcards *wc)
+{
+    uint32_t id;
+
+    memset(&wc->masks.pkt_mark, 0xff, sizeof wc->masks.pkt_mark);
+    id = recirculation_id_from_pkt_mark(flow->pkt_mark);
+    flow->pkt_mark &= ~id;
+}
diff --git a/ofproto/ofproto-dpif-recirculation.h b/ofproto/ofproto-dpif-recirculation.h
new file mode 100644
index 0000000..def28af
--- /dev/null
+++ b/ofproto/ofproto-dpif-recirculation.h
@@ -0,0 +1,81 @@
+/*
+ * Copyright (c) 2013 Nicira, Inc.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at:
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <stdbool.h>
+#include <stdint.h>
+
+#include "flow.h"
+#include "hmap.h"
+#include "ofpbuf.h"
+#include "ovs-thread.h"
+#include "tunnel.h"
+
+struct ofproto_dpif;
+struct rule_dpif;
+
+#ifndef OFPROTO_DPIF_RECIRCULATION_H
+#define OFPROTO_DPIF_RECIRCULATION_H 1
+
+#define RECIRCULATION_ID_NONE  0
+#define RECIRCULATION_ID_DUMMY 2
+#define RECIRCULATION_ID_MIN   (RECIRCULATION_ID_DUMMY + 2)
+
+struct recirculator {
+    struct hmap_node id_hmap_node; /* In recirculator_id_hmap. */
+    struct hmap_node flow_hmap_node; /* In recirculator_flow_hmap. */
+    atomic_int ref_cnt;
+
+    uint32_t id;
+    struct flow flow;
+
+    size_t ofpacts_offset;
+    size_t ofpacts_len;
+    struct ofpbuf *odp_actions;
+};
+
+static inline uint32_t recirculation_id_from_pkt_mark(uint32_t pkt_mark)
+{
+    return pkt_mark & ~IPSEC_MARK;
+}
+
+/* Check that id is a valid recirculation id */
+static inline bool recirculation_id_valid(uint32_t id)
+{
+    return id > RECIRCULATION_ID_MIN &&
+        id == recirculation_id_from_pkt_mark(id);
+}
+
+void recirculator_unref(struct recirculator *);
+void recirculator_chain_unref(struct recirculator **, size_t len);
+
+void recirculator_delete(uint32_t pkt_mark);
+uint32_t recirculation_id_get(const struct flow * ,size_t ofpacts_offset,
+                              size_t ofpacts_len,
+                              const struct ofpbuf *);
+bool recirculator_exists(uint32_t pkt_mark);
+size_t recirculator_chain_lookup(uint32_t pkt_mark,
+                                 struct recirculator **, size_t len);
+
+bool recirculator_chain_verify(struct ofproto_dpif *,
+                               struct rule_dpif *, uint8_t tcp_flags,
+                               struct recirculator **, size_t len);
+
+void recirculator_set_actions(struct recirculator *, size_t ofpacts_offset,
+                              size_t ofpacts_len, struct ofpbuf *);
+
+void recirculate_xlate_init(struct flow *, struct flow_wildcards *);
+
+#endif /* ofproto-dpif-recirculation.h */
diff --git a/ofproto/ofproto-dpif-upcall.c b/ofproto/ofproto-dpif-upcall.c
index e0a5aed..a8fcb94 100644
--- a/ofproto/ofproto-dpif-upcall.c
+++ b/ofproto/ofproto-dpif-upcall.c
@@ -28,10 +28,13 @@
 #include "latch.h"
 #include "list.h"
 #include "netlink.h"
+#include "ofp-actions.h"
 #include "ofpbuf.h"
+#include "ofproto-dpif.h"
 #include "ofproto-dpif-ipfix.h"
 #include "ofproto-dpif-sflow.h"
 #include "ofproto-dpif-xlate.h"
+#include "ofproto-dpif-recirculation.h"
 #include "packets.h"
 #include "poll-loop.h"
 #include "seq.h"
@@ -1059,7 +1062,7 @@ handle_upcalls(struct handler *handler, struct list *upcalls)
         struct xlate_in xin;
 
         xlate_in_init(&xin, miss->ofproto, &miss->flow, NULL,
-                      miss->stats.tcp_flags, NULL);
+                      miss->stats.tcp_flags, NULL, RECIRCULATION_ID_NONE);
         xin.may_learn = true;
 
         if (miss->upcall_type == DPIF_UC_MISS) {
@@ -1100,7 +1103,8 @@ handle_upcalls(struct handler *handler, struct list *upcalls)
         if (miss->xout.slow) {
             struct xlate_in xin;
 
-            xlate_in_init(&xin, miss->ofproto, &miss->flow, NULL, 0, packet);
+            xlate_in_init(&xin, miss->ofproto, &miss->flow, NULL, 0,
+                          packet, RECIRCULATION_ID_DUMMY);
             xlate_actions_for_side_effects(&xin);
         }
 
@@ -1263,6 +1267,7 @@ static bool
 revalidate_ukey(struct udpif *udpif, struct udpif_flow_dump *udump,
                 struct udpif_key *ukey)
 {
+    uint32_t pkt_mark = RECIRCULATION_ID_NONE;
     struct ofpbuf xout_actions, *actions;
     uint64_t slow_path_buf[128 / 8];
     struct xlate_out xout, *xoutp;
@@ -1310,8 +1315,10 @@ revalidate_ukey(struct udpif *udpif, struct udpif_flow_dump *udump,
     if (error) {
         goto exit;
     }
+    pkt_mark = flow.pkt_mark;
 
-    xlate_in_init(&xin, ofproto, &flow, NULL, push.tcp_flags, NULL);
+    xlate_in_init(&xin, ofproto, &flow, NULL, push.tcp_flags, NULL,
+                  RECIRCULATION_ID_DUMMY);
     xin.resubmit_stats = push.n_packets ? &push : NULL;
     xin.may_learn = push.n_packets > 0;
     xin.skip_wildcards = !udump->need_revalidate;
@@ -1357,6 +1364,7 @@ revalidate_ukey(struct udpif *udpif, struct udpif_flow_dump *udump,
 exit:
     ofpbuf_delete(actions);
     xlate_out_uninit(xoutp);
+    recirculator_delete(pkt_mark);
     return ok;
 }
 
@@ -1455,6 +1463,9 @@ revalidate_udumps(struct revalidator *revalidator, struct list *udumps)
 
     for (i = 0; i < n_ops; i++) {
         struct dpif_flow_stats push, *stats, *ukey_stats;
+        struct ofproto_dpif *ofproto;
+        struct netflow *netflow;
+        struct flow flow;
 
         ukey_stats  = &ops[i].ukey_stats;
         stats = ops[i].op.u.flow_del.stats;
@@ -1463,18 +1474,14 @@ revalidate_udumps(struct revalidator *revalidator, struct list *udumps)
         push.n_packets = stats->n_packets - ukey_stats->n_packets;
         push.n_bytes = stats->n_bytes - ukey_stats->n_bytes;
 
-        if (push.n_packets || netflow_exists()) {
-            struct ofproto_dpif *ofproto;
-            struct netflow *netflow;
-            struct flow flow;
-
-            if (!xlate_receive(udpif->backer, NULL, ops[i].op.u.flow_del.key,
-                               ops[i].op.u.flow_del.key_len, &flow,
-                               &ofproto, NULL, NULL, &netflow, NULL)) {
+        if (!xlate_receive(udpif->backer, NULL, ops[i].op.u.flow_del.key,
+                           ops[i].op.u.flow_del.key_len, &flow,
+                           &ofproto, NULL, NULL, &netflow, NULL)) {
+            if (push.n_packets || netflow_exists()) {
                 struct xlate_in xin;
 
                 xlate_in_init(&xin, ofproto, &flow, NULL, push.tcp_flags,
-                              NULL);
+                              NULL, RECIRCULATION_ID_DUMMY);
                 xin.resubmit_stats = push.n_packets ? &push : NULL;
                 xin.may_learn = push.n_packets > 0;
                 xin.skip_wildcards = true;
@@ -1486,6 +1493,8 @@ revalidate_udumps(struct revalidator *revalidator, struct list *udumps)
                     netflow_unref(netflow);
                 }
             }
+
+            recirculator_delete(flow.pkt_mark);
         }
     }
 
diff --git a/ofproto/ofproto-dpif-xlate.c b/ofproto/ofproto-dpif-xlate.c
index ffcfdf9..4d23f4f 100644
--- a/ofproto/ofproto-dpif-xlate.c
+++ b/ofproto/ofproto-dpif-xlate.c
@@ -43,6 +43,7 @@
 #include "ofproto/ofproto-dpif-ipfix.h"
 #include "ofproto/ofproto-dpif-mirror.h"
 #include "ofproto/ofproto-dpif-monitor.h"
+#include "ofproto/ofproto-dpif-recirculation.h"
 #include "ofproto/ofproto-dpif-sflow.h"
 #include "ofproto/ofproto-dpif.h"
 #include "ofproto/ofproto-provider.h"
@@ -2099,6 +2100,53 @@ execute_controller_action(struct xlate_ctx *ctx, int len,
 }
 
 static void
+compose_recirculate_action(struct xlate_ctx *ctx,
+                           const struct ofpact *ofpacts_base,
+                           const struct ofpact *ofpact_current,
+                           size_t ofpacts_base_len)
+{
+    struct flow_wildcards *wc = &ctx->xout->wc;
+    unsigned len;
+    uint32_t id;
+
+    /* The pkt_mark is part of the match of packets both before and
+     * after recirculation to allow differentiation between the two.
+     *
+     * This code sets the code for the before case.
+     * execute_flow_miss() sets the mask for the after case. */
+    memset(&wc->masks.pkt_mark, 0xff, sizeof wc->masks.pkt_mark);
+
+    ctx->xout->slow |= commit_odp_actions(&ctx->xin->flow, &ctx->base_flow,
+                                          &ctx->xout->odp_actions,
+                                          &ctx->xout->wc);
+
+    len = ofpacts_base_len - ofpact_offset(ofpacts_base, ofpact_current);
+
+    if (ctx->xin->recirculation_id != RECIRCULATION_ID_NONE) {
+        id = ctx->xin->recirculation_id;
+    } else {
+        struct rule_actions *actions;
+        unsigned offset;
+
+        actions = rule_dpif_get_actions(ctx->rule);
+        offset = ofpact_offset(actions->ofpacts, ofpact_current);
+        rule_actions_unref(actions);
+        id = recirculation_id_get(ctx->xin->orig_flow, offset, len,
+                                  &ctx->xout->odp_actions);
+    }
+
+    ctx->xin->recirculated = true;
+    ctx->xin->flow.pkt_mark |= id;
+    ctx->xin->ofpacts_len = len;
+    ctx->xin->ofpacts = ofpact_current;
+
+    ctx->xout->slow |= commit_odp_actions(&ctx->xin->flow, &ctx->base_flow,
+                                          &ctx->xout->odp_actions,
+                                          &ctx->xout->wc);
+    commit_odp_recirculate_action(&ctx->xout->odp_actions);
+}
+
+static bool
 compose_mpls_push_action(struct xlate_ctx *ctx, struct ofpact_push_mpls *mpls)
 {
     struct flow_wildcards *wc = &ctx->xout->wc;
@@ -2109,6 +2157,16 @@ compose_mpls_push_action(struct xlate_ctx *ctx, struct ofpact_push_mpls *mpls)
 
     n = flow_count_mpls_labels(flow, wc);
     if (!n) {
+        if (flow->nw_ttl == 0 &&
+            (flow->dl_type == htons(ETH_TYPE_IP) ||
+             flow->dl_type == htons(ETH_TYPE_IPV6))) {
+            /* Recirculate if it is an IP packet with a zero ttl.
+             * This may indicate that the packet was previously MPLS
+             * and an MPLS pop action converted it to IP. In this case
+             * recirculating should reveal the IP TTL which is used
+             * as the basis for a new MPLS LSE. */
+            return true;
+        }
         ctx->xout->slow |= commit_odp_actions(flow, &ctx->base_flow,
                                               &ctx->xout->odp_actions,
                                               &ctx->xout->wc);
@@ -2121,13 +2179,15 @@ compose_mpls_push_action(struct xlate_ctx *ctx, struct ofpact_push_mpls *mpls)
                          ctx->xbridge->name, FLOW_MAX_MPLS_LABELS);
         }
         ctx->exit = true;
-        return;
+        return false;
     } else if (n >= ctx->xbridge->max_mpls_depth) {
         COVERAGE_INC(xlate_actions_mpls_overflow);
         ctx->xout->slow |= SLOW_ACTION;
     }
 
     flow_push_mpls(flow, n, mpls->ethertype, wc);
+
+    return false;
 }
 
 static void
@@ -2488,6 +2548,7 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
 {
     struct flow_wildcards *wc = &ctx->xout->wc;
     struct flow *flow = &ctx->xin->flow;
+    bool may_xlate_l3_actions = true;
     const struct ofpact *a;
 
     /* dl_type already in the mask, not set below. */
@@ -2496,6 +2557,8 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
         struct ofpact_controller *controller;
         const struct ofpact_metadata *metadata;
         const struct ofpact_set_field *set_field;
+        const struct ofpact_reg_load *load;
+        const struct ofpact_reg_move *move;
         const struct mf_field *mf;
 
         if (ctx->exit) {
@@ -2526,6 +2589,9 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
 
         case OFPACT_SET_VLAN_VID:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             wc->masks.vlan_tci |= htons(VLAN_VID_MASK | VLAN_CFI);
             if (flow->vlan_tci & htons(VLAN_CFI) ||
                 ofpact_get_SET_VLAN_VID(a)->push_vlan_if_needed) {
@@ -2536,6 +2602,9 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
 
         case OFPACT_SET_VLAN_PCP:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             wc->masks.vlan_tci |= htons(VLAN_PCP_MASK | VLAN_CFI);
             if (flow->vlan_tci & htons(VLAN_CFI) ||
                 ofpact_get_SET_VLAN_PCP(a)->push_vlan_if_needed) {
@@ -2547,12 +2616,18 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
 
         case OFPACT_STRIP_VLAN:
             memset(&wc->masks.vlan_tci, 0xff, sizeof wc->masks.vlan_tci);
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             flow->vlan_tci = htons(0);
             break;
 
         case OFPACT_PUSH_VLAN:
             /* XXX 802.1AD(QinQ) */
             memset(&wc->masks.vlan_tci, 0xff, sizeof wc->masks.vlan_tci);
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             flow->vlan_tci = htons(VLAN_CFI);
             break;
 
@@ -2569,6 +2644,9 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
         case OFPACT_SET_IPV4_SRC:
             if (flow->dl_type == htons(ETH_TYPE_IP)) {
                 memset(&wc->masks.nw_src, 0xff, sizeof wc->masks.nw_src);
+                if (!may_xlate_l3_actions) {
+                    goto recirculate;
+                }
                 flow->nw_src = ofpact_get_SET_IPV4_SRC(a)->ipv4;
             }
             break;
@@ -2576,11 +2654,17 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
         case OFPACT_SET_IPV4_DST:
             if (flow->dl_type == htons(ETH_TYPE_IP)) {
                 memset(&wc->masks.nw_dst, 0xff, sizeof wc->masks.nw_dst);
+                if (!may_xlate_l3_actions) {
+                    goto recirculate;
+                }
                 flow->nw_dst = ofpact_get_SET_IPV4_DST(a)->ipv4;
             }
             break;
 
         case OFPACT_SET_IP_DSCP:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             if (is_ip_any(flow)) {
                 wc->masks.nw_tos |= IP_DSCP_MASK;
                 flow->nw_tos &= ~IP_DSCP_MASK;
@@ -2607,6 +2691,9 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             if (is_ip_any(flow)) {
                 memset(&wc->masks.nw_proto, 0xff, sizeof wc->masks.nw_proto);
                 memset(&wc->masks.tp_src, 0xff, sizeof wc->masks.tp_src);
+                if (!may_xlate_l3_actions) {
+                    goto recirculate;
+                }
                 flow->tp_src = htons(ofpact_get_SET_L4_SRC_PORT(a)->port);
             }
             break;
@@ -2615,11 +2702,17 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             if (is_ip_any(flow)) {
                 memset(&wc->masks.nw_proto, 0xff, sizeof wc->masks.nw_proto);
                 memset(&wc->masks.tp_dst, 0xff, sizeof wc->masks.tp_dst);
+                if (!may_xlate_l3_actions) {
+                    goto recirculate;
+                }
                 flow->tp_dst = htons(ofpact_get_SET_L4_DST_PORT(a)->port);
             }
             break;
 
         case OFPACT_RESUBMIT:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             xlate_ofpact_resubmit(ctx, ofpact_get_RESUBMIT(a));
             break;
 
@@ -2636,16 +2729,33 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
 
         case OFPACT_REG_MOVE:
-            nxm_execute_reg_move(ofpact_get_REG_MOVE(a), flow, wc);
+            move = ofpact_get_REG_MOVE(a);
+            mf = move->dst.field;
+
+            if (!may_xlate_l3_actions && mf_is_l3_or_higher(mf)) {
+                goto recirculate;
+            }
+            nxm_execute_reg_move(move, flow, wc);
             break;
 
         case OFPACT_REG_LOAD:
-            nxm_execute_reg_load(ofpact_get_REG_LOAD(a), flow, wc);
+            load = ofpact_get_REG_LOAD(a);
+            mf = load->dst.field;
+
+            if (!may_xlate_l3_actions && mf_is_l3_or_higher(mf)) {
+                goto recirculate;
+            }
+            nxm_execute_reg_load(load, flow, wc);
             break;
 
         case OFPACT_SET_FIELD:
             set_field = ofpact_get_SET_FIELD(a);
             mf = set_field->field;
+
+            if (!may_xlate_l3_actions && mf_is_l3_or_higher(mf)) {
+                goto recirculate;
+            }
+
             mf_mask_field_and_prereqs(mf, &wc->masks);
 
             /* Set field action only ever overwrites packet's outermost
@@ -2658,21 +2768,37 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
 
         case OFPACT_STACK_PUSH:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             nxm_execute_stack_push(ofpact_get_STACK_PUSH(a), flow, wc,
                                    &ctx->stack);
             break;
 
         case OFPACT_STACK_POP:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             nxm_execute_stack_pop(ofpact_get_STACK_POP(a), flow, wc,
                                   &ctx->stack);
             break;
 
         case OFPACT_PUSH_MPLS:
-            compose_mpls_push_action(ctx, ofpact_get_PUSH_MPLS(a));
+            if (compose_mpls_push_action(ctx, ofpact_get_PUSH_MPLS(a))) {
+                goto recirculate;
+            }
             break;
 
         case OFPACT_POP_MPLS:
+            if (!eth_type_mpls(ctx->xin->flow.dl_type)) {
+                return;
+            }
             compose_mpls_pop_action(ctx, ofpact_get_POP_MPLS(a)->ethertype);
+            if (ctx->xin->flow.dl_type == htons(ETH_TYPE_IP) ||
+                ctx->xin->flow.dl_type == htons(ETH_TYPE_IPV6) ||
+                eth_type_mpls(ctx->xin->flow.dl_type)) {
+                may_xlate_l3_actions = false;
+            }
             break;
 
         case OFPACT_SET_MPLS_LABEL:
@@ -2696,7 +2822,10 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
 
         case OFPACT_DEC_TTL:
             wc->masks.nw_ttl = 0xff;
-            if (compose_dec_ttl(ctx, ofpact_get_DEC_TTL(a))) {
+
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            } else if (compose_dec_ttl(ctx, ofpact_get_DEC_TTL(a))) {
                 return;
             }
             break;
@@ -2706,10 +2835,16 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
 
         case OFPACT_MULTIPATH:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             multipath_execute(ofpact_get_MULTIPATH(a), flow, wc);
             break;
 
         case OFPACT_BUNDLE:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             xlate_bundle_action(ctx, ofpact_get_BUNDLE(a));
             break;
 
@@ -2718,6 +2853,9 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
 
         case OFPACT_LEARN:
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
             xlate_learn_action(ctx, ofpact_get_LEARN(a));
             break;
 
@@ -2750,8 +2888,12 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
 
         case OFPACT_GOTO_TABLE: {
-            struct ofpact_goto_table *ogt = ofpact_get_GOTO_TABLE(a);
+            struct ofpact_goto_table *ogt;
 
+            if (!may_xlate_l3_actions) {
+                goto recirculate;
+            }
+            ogt = ofpact_get_GOTO_TABLE(a);
             ovs_assert(ctx->table_id < ogt->table_id);
             xlate_table_action(ctx, ctx->xin->flow.in_port.ofp_port,
                                ogt->table_id, true);
@@ -2763,15 +2905,22 @@ do_xlate_actions(const struct ofpact *ofpacts, size_t ofpacts_len,
             break;
         }
     }
+
+    return;
+
+recirculate:
+    compose_recirculate_action(ctx, ofpacts, a, ofpacts_len);
 }
 
 void
 xlate_in_init(struct xlate_in *xin, struct ofproto_dpif *ofproto,
               const struct flow *flow, struct rule_dpif *rule,
-              uint16_t tcp_flags, const struct ofpbuf *packet)
+              uint16_t tcp_flags, const struct ofpbuf *packet,
+              uint32_t recirculation_id)
 {
     xin->ofproto = ofproto;
     xin->flow = *flow;
+    xin->orig_flow = flow;
     xin->packet = packet;
     xin->may_learn = packet != NULL;
     xin->rule = rule;
@@ -2782,6 +2931,30 @@ xlate_in_init(struct xlate_in *xin, struct ofproto_dpif *ofproto,
     xin->report_hook = NULL;
     xin->resubmit_stats = NULL;
     xin->skip_wildcards = false;
+    xin->recirculation_id = recirculation_id;
+    xin->recirculated = false;
+}
+
+void
+xlate_in_init_ofpacts(struct xlate_in *xin, size_t offset, size_t len)
+{
+    struct xbridge *xbridge = xbridge_lookup(xin->ofproto);
+    struct rule_actions *actions;
+
+    /* Only use the facet's rule offset and length if the rule was found
+     * correctly, not if the miss or no packet in rule was used. */
+    if (!xbridge ||
+        xin->rule == xbridge->miss_rule ||
+        xin->rule == xbridge->no_packet_in_rule) {
+        return;
+    }
+
+    actions = rule_dpif_get_actions(xin->rule);
+    xin->ofpacts = ofpact_end(actions->ofpacts, offset);
+    rule_actions_unref(actions);
+
+    xin->ofpacts_len = len;
+
 }
 
 void
@@ -2912,6 +3085,8 @@ xlate_actions__(struct xlate_in *xin, struct xlate_out *xout)
     size_t ofpacts_len;
     bool tnl_may_send;
     bool is_icmp;
+    uint32_t recirculation_id;
+    int ofpacts_offset = 0;
 
     COVERAGE_INC(xlate_actions);
 
@@ -2973,6 +3148,13 @@ xlate_actions__(struct xlate_in *xin, struct xlate_out *xout)
         netflow_mask_wc(flow, wc);
     }
 
+    if (recirculator_exists(xin->flow.pkt_mark)) {
+        recirculation_id = recirculation_id_from_pkt_mark(xin->flow.pkt_mark);
+        recirculate_xlate_init(flow, wc);
+    } else {
+        recirculation_id = RECIRCULATION_ID_NONE;
+    }
+
     ctx.recurse = 0;
     ctx.resubmits = 0;
     ctx.orig_skb_priority = flow->skb_priority;
@@ -2980,11 +3162,46 @@ xlate_actions__(struct xlate_in *xin, struct xlate_out *xout)
     ctx.exit = false;
 
     if (!xin->ofpacts && !ctx.rule) {
-        rule_dpif_lookup(ctx.xbridge->ofproto, flow,
-                         !xin->skip_wildcards ? wc : NULL, &rule);
-        if (ctx.xin->resubmit_stats) {
+        struct flow *lookup_flow = flow;
+        bool flow_is_recirculated = recirculation_id != RECIRCULATION_ID_NONE;
+
+        struct recirculator *chain[MAX_RECIRCULATION_DEPTH + 1];
+        size_t chain_len = 0;
+
+        if (flow_is_recirculated) {
+            chain_len = recirculator_chain_lookup(recirculation_id, chain,
+                                                  ARRAY_SIZE(chain));
+            if (chain_len) {
+                lookup_flow = &chain[chain_len - 1]->flow;
+            } else {
+                /* Use the miss rule */
+                lookup_flow = NULL;
+            }
+        }
+        if (lookup_flow) {
+            rule_dpif_lookup(ctx.xbridge->ofproto, lookup_flow,
+                             !xin->skip_wildcards ? wc : NULL, &rule);
+        } else {
+            rule_dpif_choose_miss_rule(ctx.xbridge->ofproto,
+                                       flow->in_port.ofp_port, &rule);
+        }
+        if (chain_len && !rule_dpif_is_miss_rule(ctx.xbridge->ofproto, rule)) {
+            if (recirculator_chain_verify(ctx.xbridge->ofproto, rule,
+                                          xin->tcp_flags, chain, chain_len)) {
+                ofpacts_offset = chain[0]->ofpacts_offset;
+            } else {
+                /* If verification failed then use the miss rule */
+                rule_dpif_ref(rule);
+                rule_dpif_choose_miss_rule(ctx.xbridge->ofproto,
+                                           flow->in_port.ofp_port, &rule);
+            }
+        }
+        if (!flow_is_recirculated && ctx.xin->resubmit_stats) {
+            /* Do not credit statistics for recirculated packets
+             * as they were already counted before recirculation */
             rule_dpif_credit_stats(rule, ctx.xin->resubmit_stats);
         }
+        recirculator_chain_unref(chain, chain_len);
         ctx.rule = rule;
     }
     xout->fail_open = ctx.rule && rule_dpif_is_fail_open(ctx.rule);
@@ -2994,8 +3211,8 @@ xlate_actions__(struct xlate_in *xin, struct xlate_out *xout)
         ofpacts_len = xin->ofpacts_len;
     } else if (ctx.rule) {
         actions = rule_dpif_get_actions(ctx.rule);
-        ofpacts = actions->ofpacts;
-        ofpacts_len = actions->ofpacts_len;
+        ofpacts = ofpact_end(actions->ofpacts, ofpacts_offset);
+        ofpacts_len = actions->ofpacts_len - ofpacts_offset;
     } else {
         OVS_NOT_REACHED();
     }
@@ -3060,6 +3277,12 @@ xlate_actions__(struct xlate_in *xin, struct xlate_out *xout)
 
         if (tnl_may_send && (!in_port || may_receive(in_port, &ctx))) {
             do_xlate_actions(ofpacts, ofpacts_len, &ctx);
+            if (ctx.xin->recirculated) {
+                xin->ofpacts_len = ctx.xin->ofpacts_len;
+                xin->ofpacts = ctx.xin->ofpacts;
+                xin->ofpacts_len = ctx.xin->ofpacts_len;
+                xin->recirculated = ctx.xin->recirculated;
+            }
 
             /* We've let OFPP_NORMAL and the learning action look at the
              * packet, so drop it now if forwarding is disabled. */
diff --git a/ofproto/ofproto-dpif-xlate.h b/ofproto/ofproto-dpif-xlate.h
index 8b01d4e..f64d31e 100644
--- a/ofproto/ofproto-dpif-xlate.h
+++ b/ofproto/ofproto-dpif-xlate.h
@@ -61,6 +61,10 @@ struct xlate_in {
      * this flow when actions change header fields. */
     struct flow flow;
 
+    /* Original flow at the last commit prior to any commits.
+     * xlate_actions() will not modify this flow. */
+    const struct flow *orig_flow;
+
     /* The packet corresponding to 'flow', or a null pointer if we are
      * revalidating without a packet to refer to. */
     const struct ofpbuf *packet;
@@ -118,6 +122,23 @@ struct xlate_in {
      * This is normally null so the client has to set it manually after
      * calling xlate_in_init(). */
     const struct dpif_flow_stats *resubmit_stats;
+
+    /* skb_mark to use to identify recirculation.
+     *
+     * If RECIRCULATION_ID_NONE at the time that a recirculate action is
+     * added then recirculation_id_get() is used to obtain a recirculation
+     * id. Otherwise the value of this field is used.
+     *
+     * In this way both the value of the recirculation id used and the need
+     * to call recirculation_id_get() may be controlled.
+     *
+     * RECIRCULATION_ID_DUMMY may be used to provide a temporary,
+     * non-unique, recirculation id.
+     */
+    uint32_t recirculation_id;
+
+    /* True if the context added a recirculate action. False otherwise. */
+    bool recirculated;
 };
 
 extern struct ovs_rwlock xlate_rwlock;
@@ -157,16 +178,16 @@ int xlate_receive(const struct dpif_backer *, struct ofpbuf *packet,
                   struct dpif_sflow **, struct netflow **,
                   odp_port_t *odp_in_port)
     OVS_EXCLUDED(xlate_rwlock);
-
 void xlate_actions(struct xlate_in *, struct xlate_out *)
     OVS_EXCLUDED(xlate_rwlock);
 void xlate_in_init(struct xlate_in *, struct ofproto_dpif *,
                    const struct flow *, struct rule_dpif *, uint16_t tcp_flags,
-                   const struct ofpbuf *packet);
+                   const struct ofpbuf *packet,
+                   uint32_t recirculation_id);
+void xlate_in_init_ofpacts(struct xlate_in *, size_t offset, size_t len);
 void xlate_out_uninit(struct xlate_out *);
 void xlate_actions_for_side_effects(struct xlate_in *);
 void xlate_out_copy(struct xlate_out *dst, const struct xlate_out *src);
 
 int xlate_send_packet(const struct ofport_dpif *, struct ofpbuf *);
-
 #endif /* ofproto-dpif-xlate.h */
diff --git a/ofproto/ofproto-dpif.c b/ofproto/ofproto-dpif.c
index 8606226..a94f216 100644
--- a/ofproto/ofproto-dpif.c
+++ b/ofproto/ofproto-dpif.c
@@ -53,6 +53,7 @@
 #include "ofproto-dpif-ipfix.h"
 #include "ofproto-dpif-mirror.h"
 #include "ofproto-dpif-monitor.h"
+#include "ofproto-dpif-recirculation.h"
 #include "ofproto-dpif-sflow.h"
 #include "ofproto-dpif-upcall.h"
 #include "ofproto-dpif-xlate.h"
@@ -309,6 +310,9 @@ struct ofproto_dpif {
 
     /* Work queues. */
     struct guarded_list pins;      /* Contains "struct ofputil_packet_in"s. */
+
+    /* Recirculation */
+    struct hmap recirculation_ids;
 };
 
 /* All existing ofproto_dpif instances, indexed by ->up.name. */
@@ -1063,6 +1067,8 @@ construct(struct ofproto *ofproto_)
     ofproto->port_poll_errno = 0;
     ofproto->change_seq = 0;
 
+    hmap_init(&ofproto->recirculation_ids);
+
     SHASH_FOR_EACH_SAFE (node, next, &init_ofp_ports) {
         struct iface_hint *iface_hint = node->data;
 
@@ -2959,7 +2965,8 @@ ofproto_dpif_execute_actions(struct ofproto_dpif *ofproto,
         rule_dpif_credit_stats(rule, &stats);
     }
 
-    xlate_in_init(&xin, ofproto, flow, rule, stats.tcp_flags, packet);
+    xlate_in_init(&xin, ofproto, flow, rule, stats.tcp_flags, packet,
+                  RECIRCULATION_ID_DUMMY);
     xin.ofpacts = ofpacts;
     xin.ofpacts_len = ofpacts_len;
     xin.resubmit_stats = &stats;
@@ -3032,6 +3039,14 @@ rule_dpif_get_actions(const struct rule_dpif *rule)
 }
 
 /* Choose the miss rule for 'in_port' in 'ofproto'. */
+bool
+rule_dpif_is_miss_rule(const struct ofproto_dpif *ofproto,
+                       const struct rule_dpif *rule)
+{
+    return rule == ofproto->miss_rule || rule == ofproto->no_packet_in_rule;
+}
+
+/* Choose the miss rule for 'in_port' in 'ofproto'. */
 void
 rule_dpif_choose_miss_rule(struct ofproto_dpif *ofproto, ofp_port_t in_port,
                            struct rule_dpif **rule)
@@ -3070,7 +3085,7 @@ rule_dpif_lookup_in_table(struct ofproto_dpif *ofproto,
     bool frag;
 
     *rule = NULL;
-    if (table_id >= N_TABLES) {
+    if (!flow || table_id >= N_TABLES) {
         return false;
     }
 
@@ -3936,7 +3951,8 @@ ofproto_trace(struct ofproto_dpif *ofproto, const struct flow *flow,
         tcp_flags = packet ? packet_get_tcp_flags(packet, flow) : 0;
         trace.result = ds;
         trace.flow = *flow;
-        xlate_in_init(&trace.xin, ofproto, flow, rule, tcp_flags, packet);
+        xlate_in_init(&trace.xin, ofproto, flow, rule, tcp_flags, packet,
+                      RECIRCULATION_ID_DUMMY);
         if (ofpacts) {
             trace.xin.ofpacts = ofpacts;
             trace.xin.ofpacts_len = ofpacts_len;
diff --git a/ofproto/ofproto-dpif.h b/ofproto/ofproto-dpif.h
index c0fedee..a468ae8 100644
--- a/ofproto/ofproto-dpif.h
+++ b/ofproto/ofproto-dpif.h
@@ -67,8 +67,11 @@ extern struct ovs_rwlock xlate_rwlock;
 
 size_t ofproto_dpif_get_max_mpls_depth(const struct ofproto_dpif *);
 
+bool
+rule_dpif_is_miss_rule(const struct ofproto_dpif *, const struct rule_dpif *);
+
 void
-rule_dpif_choose_miss_rule(struct ofproto_dpif *ofproto, ofp_port_t in_port,
+rule_dpif_choose_miss_rule(struct ofproto_dpif *, ofp_port_t in_port,
                            struct rule_dpif **rule);
 
 void rule_dpif_lookup(struct ofproto_dpif *, const struct flow *,
@@ -124,5 +127,4 @@ int ofproto_dpif_send_packet(const struct ofport_dpif *, struct ofpbuf *);
 void ofproto_dpif_flow_mod(struct ofproto_dpif *, struct ofputil_flow_mod *);
 
 struct ofport_dpif *odp_port_to_ofport(const struct dpif_backer *, odp_port_t);
-
 #endif /* ofproto-dpif.h */
diff --git a/ofproto/tunnel.c b/ofproto/tunnel.c
index 09497a3..1b1d3e5 100644
--- a/ofproto/tunnel.c
+++ b/ofproto/tunnel.c
@@ -33,9 +33,6 @@
 
 VLOG_DEFINE_THIS_MODULE(tunnel);
 
-/* skb mark used for IPsec tunnel packets */
-#define IPSEC_MARK 1
-
 struct tnl_match {
     ovs_be64 in_key;
     ovs_be32 ip_src;
diff --git a/ofproto/tunnel.h b/ofproto/tunnel.h
index 27a2f7d..afe78ab 100644
--- a/ofproto/tunnel.h
+++ b/ofproto/tunnel.h
@@ -20,6 +20,9 @@
 #include <stdint.h>
 #include "flow.h"
 
+/* skb mark used for IPsec tunnel packets */
+#define IPSEC_MARK 1
+
 /* Tunnel port emulation layer.
  *
  * These functions emulate tunnel virtual ports based on the outer
diff --git a/tests/ofproto-dpif.at b/tests/ofproto-dpif.at
index 35d8c79..bca44de 100644
--- a/tests/ofproto-dpif.at
+++ b/tests/ofproto-dpif.at
@@ -712,9 +712,38 @@ cookie=0xa dl_src=40:44:44:44:44:45 actions=push_mpls:0x8847,load:10->OXM_OF_MPL
 cookie=0xa dl_src=40:44:44:44:44:46 actions=push_mpls:0x8847,load:10->OXM_OF_MPLS_LABEL[[]],load:3->OXM_OF_MPLS_TC[[]],set_mpls_ttl(10),controller
 cookie=0xa dl_src=40:44:44:44:44:47 actions=push_mpls:0x8847,load:10->OXM_OF_MPLS_LABEL[[]],load:3->OXM_OF_MPLS_TC[[]],dec_mpls_ttl,set_mpls_ttl(10),controller
 cookie=0xa dl_src=40:44:44:44:44:48 actions=push_mpls:0x8847,load:10->OXM_OF_MPLS_LABEL[[]],load:3->OXM_OF_MPLS_TC[[]],set_mpls_ttl(10),dec_mpls_ttl,controller
+cookie=0xa mpls,dl_src=40:44:44:44:44:49 actions=push_mpls:0x8848,load:10->OXM_OF_MPLS_LABEL[[]],CONTROLLER:65535
 cookie=0xb dl_src=50:55:55:55:55:55 dl_type=0x8847 actions=load:1000->OXM_OF_MPLS_LABEL[[]],controller
 cookie=0xd dl_src=60:66:66:66:66:66 actions=pop_mpls:0x0800,controller
 cookie=0xc dl_src=70:77:77:77:77:77 actions=push_mpls:0x8848,load:1000->OXM_OF_MPLS_LABEL[[]],load:7->OXM_OF_MPLS_TC[[]],controller
+
+cookie=0xd dl_src=60:66:66:66:00:01 actions=pop_mpls:0x0800,dec_ttl,controller
+cookie=0xd dl_src=60:66:66:66:00:02 actions=pop_mpls:0x0800,load:0xa000001->OXM_OF_IPV4_DST[[]],controller
+cookie=0xd dl_src=60:66:66:66:00:03 actions=pop_mpls:0x0800,move:OXM_OF_IPV4_DST[[]]->OXM_OF_IPV4_SRC[[]],controller
+cookie=0xd dl_src=60:66:66:66:00:04 actions=pop_mpls:0x0800,push:OXM_OF_IPV4_DST[[]],pop:OXM_OF_IPV4_SRC[[]],controller
+cookie=0xd dl_src=60:66:66:66:00:05 actions=pop_mpls:0x0800,multipath(eth_src,50,modulo_n,256,0,OXM_OF_IPV4_SRC[[0..7]]),controller
+cookie=0xd dl_src=60:66:66:66:00:06 actions=pop_mpls:0x0800,bundle_load(eth_src,50,hrw,ofport,OXM_OF_IPV4_SRC[[0..15]],slaves:1,2),controller
+cookie=0xd dl_src=60:66:66:66:00:07 actions=pop_mpls:0x0800,learn(table=1,hard_timeout=60,eth_type=0x800,nw_proto=6,OXM_OF_IPV4_SRC[[]]=OXM_OF_IPV4_DST[[]]),controller
+
+cookie=0xd dl_src=60:66:66:66:01:00 actions=pop_mpls:0x8848,controller
+cookie=0xd dl_src=60:66:66:66:01:01 actions=pop_mpls:0x8847,dec_mpls_ttl,controller
+cookie=0xd dl_src=60:66:66:66:01:02 actions=pop_mpls:0x8848,load:3->OXM_OF_MPLS_TC[[]],controller
+
+cookie=0xd dl_src=60:66:66:66:02:00 actions=pop_mpls:0x8847,pop_mpls:0x0800,controller
+cookie=0xd dl_src=60:66:66:66:02:01 actions=pop_mpls:0x8848,pop_mpls:0x0800,dec_ttl,controller
+cookie=0xd dl_src=60:66:66:66:02:10 actions=pop_mpls:0x8847,dec_mpls_ttl,pop_mpls:0x0800,dec_ttl,controller
+
+cookie=0xd dl_src=60:66:66:66:03:00 actions=pop_mpls:0x8848,pop_mpls:0x8848,controller
+cookie=0xd dl_src=60:66:66:66:03:01 actions=pop_mpls:0x8847,pop_mpls:0x8847,dec_mpls_ttl,controller
+cookie=0xd dl_src=60:66:66:66:03:10 actions=pop_mpls:0x8848,dec_mpls_ttl,pop_mpls:0x8848,dec_mpls_ttl,controller
+
+cookie=0xd dl_src=60:66:66:66:04:00 actions=pop_mpls:0x0800,push_mpls:0x8847,controller
+cookie=0xd dl_src=60:66:66:66:04:01 actions=pop_mpls:0x0800,push_mpls:0x8848,dec_mpls_ttl,controller
+cookie=0xd dl_src=60:66:66:66:04:10 actions=pop_mpls:0x0800,dec_ttl,push_mpls:0x8848,dec_mpls_ttl,controller
+
+cookie=0xd dl_src=60:66:66:66:05:00 actions=push_mpls:0x8848,pop_mpls:0x8847,controller
+cookie=0xd dl_src=60:66:66:66:05:01 actions=push_mpls:0x8847,pop_mpls:0x8848,dec_mpls_ttl,controller
+cookie=0xd dl_src=60:66:66:66:05:10 actions=push_mpls:0x8848,dec_mpls_ttl,pop_mpls:0x8847,dec_mpls_ttl,controller
 ])
 AT_CHECK([ovs-ofctl add-flows br0 flows.txt])
 
@@ -863,6 +892,26 @@ dnl Modified MPLS controller action.
 AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
 
 for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 'in_port(1),eth(src=40:44:44:44:44:49,dst=50:54:00:00:00:07),eth_type(0x8847),mpls(label=10,tc=3,ttl=64,bos=1)'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xa total_len=64 in_port=1 (via action) data_len=64 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=40:44:44:44:44:49,dl_dst=50:54:00:00:00:07,mpls_label=10,mpls_tc=3,mpls_ttl=64,mpls_bos=0,mpls_lse1=42816
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xa total_len=64 in_port=1 (via action) data_len=64 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=40:44:44:44:44:49,dl_dst=50:54:00:00:00:07,mpls_label=10,mpls_tc=3,mpls_ttl=64,mpls_bos=0,mpls_lse1=42816
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xa total_len=64 in_port=1 (via action) data_len=64 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=40:44:44:44:44:49,dl_dst=50:54:00:00:00:07,mpls_label=10,mpls_tc=3,mpls_ttl=64,mpls_bos=0,mpls_lse1=42816
+])
+
+dnl Modified MPLS controller action.
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
     ovs-appctl netdev-dummy/receive p1 'in_port(1),eth(src=40:44:44:44:44:48,dst=50:54:00:00:00:07),eth_type(0x0800),ipv4(src=192.168.0.1,dst=192.168.0.2,proto=16,tos=0,ttl=64,frag=no)'
 done
 OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
@@ -947,6 +996,546 @@ NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len
 tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:66:66,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=64,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
 ])
 
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:00:01 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8847), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 00 01 88 48 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:01,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:01,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:01,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:00:02 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8847), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 00 02 88 47 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:02,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:2dee
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:02,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:2dee
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:02,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=10.0.0.1,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:2dee
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:00:03 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8848), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 00 03 88 48 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:03,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.2,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7743
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:03,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.2,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7743
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:03,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.2,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7743
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:00:04 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8847), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 00 04 88 47 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:04,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.2,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7743
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:04,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.2,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7743
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:04,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.2,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7743
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:00:05 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8848), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 00 05 88 48 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.106,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:76db
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.106,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:76db
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:05,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.106,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:76db
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:00:06 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8847), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 00 06 88 47 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:06,dl_dst=50:54:00:00:00:07,nw_src=192.168.255.255,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7745
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:06,dl_dst=50:54:00:00:00:07,nw_src=192.168.255.255,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7745
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:06,dl_dst=50:54:00:00:00:07,nw_src=192.168.255.255,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7745
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:00:07 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8848), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 00 07 88 48 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:07,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:07,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:00:07,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with two MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:01:00 > 50:54:00:00:00:07, ethertype MPLS multicast (0x8848), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, [S], ttl 31)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 01 00 88 48 00 01 40 20 00 01 41 1f 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with two MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:01:01 > 50:54:00:00:00:07, ethertype MPLS multicast (0x8847), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, [S], ttl 31)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 01 01 88 47 00 01 40 20 00 01 41 1f 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=30,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=30,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=30,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with two MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:01:02 > 50:54:00:00:00:07, ethertype MPLS multicast (0x8848), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, [S], ttl 31)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 01 02 88 48 00 01 40 20 00 01 41 1f 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:02,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=3,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:02,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=3,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:01:02,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=3,mpls_ttl=31,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with two MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:02:00 > 50:54:00:00:02:00, ethertype MPLS multicast (0x8847), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, [S], ttl 31)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 02 00 88 47 00 01 40 20 00 01 41 1f 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:00,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:00,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:00,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=255,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with two MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:02:01 > 50:54:00:00:02:01, ethertype MPLS multicast (0x8848), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, [S], ttl 31)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 02 01 88 48 00 01 40 20 00 01 41 1f 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:01,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:01,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:01,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with two MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:02:10 > 50:54:00:00:02:10, ethertype MPLS multicast (0x8847), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, [S], ttl 31)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 02 10 88 47 00 01 40 20 00 01 41 1f 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:10,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:10,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=58 in_port=1 (via action) data_len=58 (unbuffered)
+tcp,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:02:10,dl_dst=50:54:00:00:00:07,nw_src=192.168.0.1,nw_dst=192.168.0.2,nw_tos=0,nw_ecn=0,nw_ttl=254,tp_src=80,tp_dst=0,tcp_flags=0x000 tcp_csum:7744
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with three MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:03:00 > 50:54:00:00:00:00, ethertype MPLS multicast (0x8847), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, ttl 31)
+dnl		(label 20, exp 0, [S], ttl 30)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 03 00 88 47 00 01 40 20 00 01 40 1f 00 01 41 1e 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=30,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=30,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=30,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with three MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:03:01 > 50:54:00:00:00:00, ethertype MPLS multicast (0x8848), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, ttl 31)
+dnl		(label 20, exp 0, [S], ttl 30)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 03 01 88 48 00 01 40 20 00 01 40 1f 00 01 41 1e 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=29,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=29,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=29,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with three MPLS label stack entries which tcpdump -vve shows as:
+dnl 60:66:66:66:03:10 > 50:54:00:00:00:00, ethertype MPLS multicast (0x8847), length 66: MPLS (label 20, exp 0, ttl 32)
+dnl		(label 20, exp 0, ttl 31)
+dnl		(label 20, exp 0, [S], ttl 30)
+dnl		(tos 0x0, ttl 254, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl        192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x7744 (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 03 10 88 47 00 01 40 20 00 01 40 1f 00 01 41 1e 45 00 00 2c 00 00 00 00 ff 06 3b 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:10,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=29,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:10,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=29,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:03:10,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=29,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:04:00 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8848), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 04 00 88 48 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:00,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=255,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:00,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=255,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:00,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=255,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:04:01 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8847), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 04 01 88 47 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:01,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=254,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:01,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=254,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:01,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=254,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:04:10 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8848), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 04 10 88 48 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:10,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=253,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:10,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=253,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:04:10,dl_dst=50:54:00:00:00:07,mpls_label=0,mpls_tc=0,mpls_ttl=253,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:05:00 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8847), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 05 00 88 47 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=32,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=32,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:00,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=32,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:05:01 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8848), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 05 01 88 48 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mplsm,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:01,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+])
+
+dnl Modified MPLS pop action.
+dnl The input is a frame with a single MPLS label stack entry which tcpdump -vve shows as:
+dnl 60:66:66:66:05:10 > 50:54:00:00:00:07, ethertype MPLS unicast (0x8847), length 62: MPLS (label 20, exp 0, [S], ttl 32)
+dnl             (tos 0x0, ttl 255, id 0, offset 0, flags [none], proto TCP (6), length 44)
+dnl         192.168.0.1.80 > 192.168.0.2.0: Flags [none], cksum 0x77ec (correct), seq 42:46, win 10000, length 4
+AT_CHECK([ovs-ofctl monitor br0 65534 -P nxm --detach --pidfile 2> ofctl_monitor.log])
+
+for i in 1 2 3; do
+    ovs-appctl netdev-dummy/receive p1 '50 54 00 00 00 07 60 66 66 66 05 10 88 47 00 01 41 20 45 00 00 2c 00 00 00 00 ff 06 3a 78 c0 a8 00 01 c0 a8 00 02 00 50 00 00 00 00 00 2a 00 00 00 2a 50 00 27 10 77 44 00 00 48 4f 47 45'
+done
+OVS_WAIT_UNTIL([test `wc -l < ofctl_monitor.log` -ge 6])
+ovs-appctl -t ovs-ofctl exit
+
+AT_CHECK([cat ofctl_monitor.log], [0], [dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:10,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:10,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+dnl
+NXT_PACKET_IN (xid=0x0): cookie=0xd total_len=62 in_port=1 (via action) data_len=62 (unbuffered)
+mpls,metadata=0,in_port=0,vlan_tci=0x0000,dl_src=60:66:66:66:05:10,dl_dst=50:54:00:00:00:07,mpls_label=20,mpls_tc=0,mpls_ttl=31,mpls_bos=1
+])
+
 AT_CHECK([ovs-appctl time/warp 5000], [0], [ignore])
 
 AT_CHECK([ovs-ofctl dump-flows br0 | ofctl_strip | sort], [0], [dnl
@@ -958,9 +1547,33 @@ AT_CHECK([ovs-ofctl dump-flows br0 | ofctl_strip | sort], [0], [dnl
  cookie=0xa, n_packets=3, n_bytes=180, dl_src=40:44:44:44:44:47 actions=push_mpls:0x8847,load:0xa->OXM_OF_MPLS_LABEL[[]],load:0x3->OXM_OF_MPLS_TC[[]],dec_mpls_ttl,set_mpls_ttl(10),CONTROLLER:65535
  cookie=0xa, n_packets=3, n_bytes=180, dl_src=40:44:44:44:44:48 actions=push_mpls:0x8847,load:0xa->OXM_OF_MPLS_LABEL[[]],load:0x3->OXM_OF_MPLS_TC[[]],set_mpls_ttl(10),dec_mpls_ttl,CONTROLLER:65535
  cookie=0xa, n_packets=3, n_bytes=180, dl_src=41:44:44:44:44:42 actions=push_mpls:0x8847,load:0xa->OXM_OF_MPLS_LABEL[[]],load:0x3->OXM_OF_MPLS_TC[[]],pop_mpls:0x0800,CONTROLLER:65535
+ cookie=0xa, n_packets=3, n_bytes=180, mpls,dl_src=40:44:44:44:44:49 actions=push_mpls:0x8848,load:0xa->OXM_OF_MPLS_LABEL[[]],CONTROLLER:65535
  cookie=0xb, n_packets=3, n_bytes=180, mpls,dl_src=50:55:55:55:55:55 actions=load:0x3e8->OXM_OF_MPLS_LABEL[[]],CONTROLLER:65535
  cookie=0xc, n_packets=3, n_bytes=180, dl_src=70:77:77:77:77:77 actions=push_mpls:0x8848,load:0x3e8->OXM_OF_MPLS_LABEL[[]],load:0x7->OXM_OF_MPLS_TC[[]],CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:00:01 actions=pop_mpls:0x0800,dec_ttl(0),CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:00:02 actions=pop_mpls:0x0800,load:0xa000001->NXM_OF_IP_DST[[]],CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:00:03 actions=pop_mpls:0x0800,move:NXM_OF_IP_DST[[]]->NXM_OF_IP_SRC[[]],CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:00:04 actions=pop_mpls:0x0800,push:NXM_OF_IP_DST[[]],pop:NXM_OF_IP_SRC[[]],CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:00:05 actions=pop_mpls:0x0800,multipath(eth_src,50,modulo_n,256,0,NXM_OF_IP_SRC[[0..7]]),CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:00:06 actions=pop_mpls:0x0800,bundle_load(eth_src,50,hrw,ofport,NXM_OF_IP_SRC[[0..15]],slaves:1,2),CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:00:07 actions=pop_mpls:0x0800,learn(table=1,hard_timeout=60,eth_type=0x800,nw_proto=6,NXM_OF_IP_SRC[[]]=NXM_OF_IP_DST[[]]),CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:04:00 actions=pop_mpls:0x0800,push_mpls:0x8847,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:04:01 actions=pop_mpls:0x0800,push_mpls:0x8848,dec_mpls_ttl,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:04:10 actions=pop_mpls:0x0800,dec_ttl(0),push_mpls:0x8848,dec_mpls_ttl,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:05:00 actions=push_mpls:0x8848,pop_mpls:0x8847,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:05:01 actions=push_mpls:0x8847,pop_mpls:0x8848,dec_mpls_ttl,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:05:10 actions=push_mpls:0x8848,dec_mpls_ttl,pop_mpls:0x8847,dec_mpls_ttl,CONTROLLER:65535
  cookie=0xd, n_packets=3, n_bytes=186, dl_src=60:66:66:66:66:66 actions=pop_mpls:0x0800,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=198, dl_src=60:66:66:66:01:00 actions=pop_mpls:0x8848,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=198, dl_src=60:66:66:66:01:01 actions=pop_mpls:0x8847,dec_mpls_ttl,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=198, dl_src=60:66:66:66:01:02 actions=pop_mpls:0x8848,load:0x3->OXM_OF_MPLS_TC[[]],CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=198, dl_src=60:66:66:66:02:00 actions=pop_mpls:0x8847,pop_mpls:0x0800,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=198, dl_src=60:66:66:66:02:01 actions=pop_mpls:0x8848,pop_mpls:0x0800,dec_ttl(0),CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=198, dl_src=60:66:66:66:02:10 actions=pop_mpls:0x8847,dec_mpls_ttl,pop_mpls:0x0800,dec_ttl(0),CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=210, dl_src=60:66:66:66:03:00 actions=pop_mpls:0x8848,pop_mpls:0x8848,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=210, dl_src=60:66:66:66:03:01 actions=pop_mpls:0x8847,pop_mpls:0x8847,dec_mpls_ttl,CONTROLLER:65535
+ cookie=0xd, n_packets=3, n_bytes=210, dl_src=60:66:66:66:03:10 actions=pop_mpls:0x8848,dec_mpls_ttl,pop_mpls:0x8848,dec_mpls_ttl,CONTROLLER:65535
+ table=1, hard_timeout=60, tcp,nw_src=192.168.0.2 actions=drop
 NXST_FLOW reply:
 ])
 
diff --git a/utilities/ovs-ofctl.8.in b/utilities/ovs-ofctl.8.in
index 9a8fd33..8cbce0b 100644
--- a/utilities/ovs-ofctl.8.in
+++ b/utilities/ovs-ofctl.8.in
@@ -1228,11 +1228,6 @@ if \fBpush_mpls\fR follows another \fBpush_mpls\fR unless there is a
 .
 .IP \fBpop_mpls\fR:\fIethertype\fR
 Strips the outermost MPLS label stack entry.
-Currently the implementation restricts \fIethertype\fR to a non-MPLS Ethertype
-and thus \fBpop_mpls\fR should only be applied to packets with
-an MPLS label stack depth of one. A further limitation is that processing of
-actions will stop if \fBpop_mpls\fR follows another \fBpop_mpls\fR unless
-there is a \fBpush_mpls\fR in between.
 .
 .IP \fBmod_dl_src\fB:\fImac\fR
 Sets the source Ethernet address to \fImac\fR.
-- 
1.8.5.2




More information about the dev mailing list