[ovs-dev] [ovs-dev, RFC] ovn: Revised support for service function chaining

Mickey Spiegel mickeys.dev at gmail.com
Thu Mar 16 01:24:09 UTC 2017


On Mon, Mar 13, 2017 at 1:28 PM, John McDowall <
jmcdowall at paloaltonetworks.com> wrote:

> This patch set is an alternative implementation of service function
> chaining (SFC) for OVS/OVN. The major change from the previous patch is
> that the overloading of the ACL stage in ovn-northd.c has been removed
> and replaced with additional logic in the CHAIN stage.
>
> This was done to improve modularity of the code as it was felt that
> overloading the ACL stage did not add a lot and made it hard to compose
> reusable SFCs.
>
> The new approach can still use the matching logic in OVN as a match
> argument has been added. This has not been fully implemented yet as it
> may need some error checking to ensure the match does not conflicit
> with the lport in the chain and the bi-directional case.
>
> The other major change is the logic for the final destination of the
> flows exiting the service chain. This has been simplified such that the
> rules in the chain stage determine when the flow is exiting the port-chain
> and then the flow just follows the normal path to the src or dst of the
> flow.
>
> Areas to review
>
> 1) Logic for delivering flow after it leaves the chain. I think it is now
> general and should work across subnets etc.
>

This revision does address one of my concerns with the previous revision,
specifically the previous proposal to specify a 'last_hop_port' at the end
of the logical port chain.

However, it does not address my other major concern about the behavior at
the end of the service chain, the interaction with the way OVN forwards
traffic to the outside world.


Repeating my comments from the last revision:

In OVN without SFC, traffic is forwarded through a logical switch to a
logical switch port of type 'localnet' that effectively resides on all
hypervisors, that then forwards to the physical L2 network.  When a VM
residing on that same logical switch originates traffic destined for the
outside world, it is directed to the local instance of the type 'localnet'
logical switch port on the VM's hypervisor.  The physical L2 network learns
that the VM's MAC address resides on that hypervisor, and forwards all
traffic to that MAC address to that specific hypervisor.  Typically the
physical L2 network first learns the VM's MAC address due to a gratuitous
ARP packet sent by the ovn-controller on the VM's hypervisor.

In this SFC proposal, at the end of the logical port chain, traffic is
forwarded directly to the destination.  This might work for destinations
attached directly to OVN, but it would break forwarding to the physical L2
network:

1. If the last port pair group in the logical port chain contains more than
one port pair spread across more than one chassis, then this would
completely break upstream L2 learning.  Traffic would have to be forwarded
to a common point before being sent upstream.

2. What if the SFC classifier granularity is finer than per source VM, and
the multiple logical port chains (that one VM's traffic can be redirected
to) end at different chassis?

3. The gratuitous ARP packets generated by OVN for VIFs residing on the
same logical switch as the 'localnet' port are sent from each VIF's
chassis.  This may be different than the chassis at the end of the logical
port chain.

There are some topologies where this SFC proposal will work even for
traffic destined for the outside world: if there are no VIFs on the same
logical switch as the 'localnet' port, and either a gateway router is used
or a distributed router with only centralized NAT rules is used.  If a
distributed router is specified with distributed NAT rules, then point 3
above gets even worse, since ARP replies for a distributed NAT external IP
address are restricted to the chassis where the corresponding VIF resides.

My preference is to return to the originating hypervisor at the end of the
logical port chain.  With such an approach, L2 and L3 forwarding are
unaffected.
However, there is a question how to determine from a packet at the end of
the logical port chain, what is the originating hypervisor?
The best answer is to use an encapsulation such as Geneve or NSH that can
carry context information.
Another alternative is to do a lookup of the source address when the chain
was entered in the "exit-lport" direction.


An example to clarify the problem:

Single Logical Switch LS1.
VM1 on HV1 with MAC1, IP1.
Logical Port Chain LPC1 with one Logical Port Pair Group LPPG1.
LPPG1 has two Logical Port Pairs LPP1 and LPP2, residing on HV2 and HV3
respectively.

When VM1 comes up, HV1 will send a gratuitous ARP packet out the localnet
interface on HV1 to the physical L2 network, using MAC1 and IP1.
The physical L2 network will learn that MAC1 is on HV1.

VM1 sends a packet to the outside world.
The packet gets redirected down LPC1 to LPP1 on HV2. Using the logic below,
the packet will go through a normal L2 destination lookup on HV2, so it
will be sent out the localnet interface on HV2 to the physical L2 network.
The physical L2 network will now learn that MAC1 is on HV2.

VM1 sends a packet for a different flow to the outside world.
The packet gets redirected down LPC1 to LPP2 on HV3. Using the logical
below, the packet will go through a normal L2 destination lookup on HV3, so
it will be sent out the localnet interface on HV3 to the physical L2
network.
The physical L2 network will now learn that MAC1 is on HV3.


2) Do the command names make sense - currently rather long and complex.
> 3) Using the match to classify flows to a finer granularity is mainly
> useful for uni-directional flows but could lead to conflicits with
> bi-directional flows (I think) and suggestions on how to make this better
> would be appreciated.
>

I wonder about the implicit classification in this patch.

If I am reading the code correctly, when a user specifies a "port" in a
"Logical_Port_Chain_Classifier", that port is not used directly as match
criteria. Instead, the first IP address of the port is used in match
criteria, not just for the initial classification into the logical port
chain, but also for subsequent hops along the port chain.

The primary benefit of this approach is that it allows a VNF to be used in
different logical port chains. Since the IP address is used in the match
criteria, different logical port chain classifiers on different ports with
different logical port chains can use the same logical port pair.

This approach also allows for "entry-lport" logical port chain classifiers
to be applied in the ingress pipeline, before the egress port is determined.

I can think of several issues with this approach:

a) If two different logical port chain classifiers with different "match"
were specified on the same port, with different logical port chains that
share at least one logical port pair, then the IP address does not
distinguish between the two logical port chains.

b) This approach does not support non-IP traffic.

c) This approach does not support VNFs that modify IP addresses such as
load balancers or NAT.

Another issue is that OVN allows a port to have multiple LSP addresses,
each specifying any number of IP addresses. This can be fixed in the code
below.

In terms of interaction with "match", in general it should not be a
problem. Just concatenate (" && %s", implicit match criteria) with the user
specified "match". For bidirectional logical port chains, it would be up to
the user to determine that the "match" criteria makes sense, e.g. does not
include any "source" or "destination" matches. Otherwise the user should
use "uni-directional" logical port chains.


> Current todo list
>
> 1) I have standalone tests need to add tests to ovs/ovn framework.
> 2) Add IPv6 support
> 3) Implement "match" in the CHAIN stage.
> 4) Load-balancing support for port-pair-groups
> 5) Publish more detailed examples.
>
> Simple test case using ovn-trace
>
> #!/bin/sh
> #
> clear
> ovn-nbctl ls-add swt1
>
> ovn-nbctl lsp-add swt1 swt1-appc
> ovn-nbctl lsp-add swt1 swt1-apps
> ovn-nbctl lsp-add swt1 swt1-vnfp1
> ovn-nbctl lsp-add swt1 swt1-vnfp2
>
> ovn-nbctl lsp-set-addresses swt1-appc "00:00:00:00:00:01 192.168.33.1"
> ovn-nbctl lsp-set-addresses swt1-apps "00:00:00:00:00:02 192.168.33.2"
> ovn-nbctl lsp-set-addresses swt1-vnfp1 00:00:00:00:00:03
> ovn-nbctl lsp-set-addresses swt1-vnfp2 00:00:00:00:00:04
> #
> # Configure Service chain
> #
> ovn-nbctl lsp-pair-add swt1 swt1-vnfp1 swt1-vnfp2 pp1
> ovn-nbctl lsp-chain-add swt1 pc1
> ovn-nbctl lsp-pair-group-add pc1 ppg1
> ovn-nbctl lsp-pair-group-add-port-pair ppg1 pp1
> ovn-nbctl lsp-chain-classifier-add swt1 pc1 swt1-appc "exit-lport"
> "bi-directional" "" pcc1
> #
> ovn-sbctl dump-flows
> #
> # Run trace command
> printf "\n---------Flow 1 -------------\n\n"
> ovn-trace --detailed  swt1 'inport == "swt1-appc" && eth.src ==
> 00:00:00:00:00:01 && eth.dst == 00:00:00:00:00:02 && ip4.src ==
> 192.168.33.1 && ip4.dst == 192.168.33.2'
> printf "\n---------Flow 2 -------------\n\n"
> ovn-trace --detailed  swt1 'inport == "swt1-vnfp1" && eth.src ==
> 00:00:00:00:00:01 && eth.dst == 00:00:00:00:00:02 && ip4.src ==
> 192.168.33.1 && ip4.dst == 192.168.33.2'
> printf "\n---------Flow 3 -------------\n\n"
> ovn-trace --detailed  swt1 'inport == "swt1-apps" && eth.dst ==
> 00:00:00:00:00:01 && eth.src == 00:00:00:00:00:02 && ip4.dst ==
> 192.168.33.1 && ip4.src == 192.168.33.2'
> printf "\n---------Flow 4 -------------\n\n"
> ovn-trace --detailed  swt1 'inport == "swt1-vnfp2" && eth.dst ==
> 00:00:00:00:00:01 && eth.src == 00:00:00:00:00:02 && ip4.dst ==
> 192.168.33.1 && ip4.src == 192.168.33.2'
> #
> # Cleanup
> #
> ovn-nbctl lsp-chain-classifier-del pcc1
> ovn-nbctl lsp-pair-group-del ppg1
> ovn-nbctl lsp-chain-del pc1
> ovn-nbctl lsp-pair-del pp1
> ovn-nbctl ls-del swt1
>
> Co-authored-by: Flavio Fernandes <flavio at flaviof.com>
> Reported at: https://mail.openvswitch.org/pipermail/ovs-discuss/2016-
> March/040381.html
> Reported at: https://mail.openvswitch.org/pipermail/ovs-discuss/2016-
> May/041359.html
>
> Signed-off-by: John McDowall <jmcdowall at paloaltonetworks.com>
> ---
>  ovn/northd/ovn-northd.8.xml   |   60 ++-
>  ovn/northd/ovn-northd.c       |  293 ++++++++++-
>  ovn/ovn-architecture.7.xml    |   91 ++++
>  ovn/ovn-nb.ovsschema          |   87 +++-
>  ovn/ovn-nb.xml                |  187 +++++++
>  ovn/utilities/ovn-nbctl.8.xml |  218 ++++++++
>  ovn/utilities/ovn-nbctl.c     | 1133 ++++++++++++++++++++++++++++++
> +++++++++++
>  7 files changed, 2042 insertions(+), 27 deletions(-)
>
> diff --git ovn/northd/ovn-northd.8.xml ovn/northd/ovn-northd.8.xml
> index ab8fd88..f80fcf6 100644
> --- ovn/northd/ovn-northd.8.xml
> +++ ovn/northd/ovn-northd.8.xml
> @@ -362,7 +362,53 @@
>        </li>
>      </ul>
>
> -    <h3>Ingress Table 7: <code>from-lport</code> QoS marking</h3>
> +     <h3>Ingress Table 7: <code>from-lport</code> Port Chaining</h3>
> +
> +    <p>
> +      Logical flows in this table closely reproduce those in the
> +      <code>QoS</code> table in the <code>OVN_Northbound</code> database
> +      for the <code>from-lport</code> direction.
> +    </p>
> +
> +    <ul>
> +      <li>
> +        For every port-chain a set of rules will be added to direct
> traffic
> +        through the port pairs defined in the port-chain. A port chain
> +        is composed of an ordered set of port-pair-groups that contain
> one or
> +        more port-pairs. Traffic is directed into the port-chain by
> creating a
> +        port-chain-classifier. A port-chain can be reused by different
> +        port-chain-classifier instances allowing a port chain to be
> +        applied to multiple traffic paths and application traffic types.
> +
> +        The port-chain-classifier defines a starting port or ending port
> and
> +        a direction for the traffic, either uni-directional or
> bi-directional.
> +        In addition a match expression can be defined to further filter
> +        traffic.
> +
> +        Service insertion is implemented by adding 4 new flow rules into
> the
> +        OVN northbound database for each VNF inserted. The service
> +        insertion rules have a higher priority than the standard
> forwarding
> +        rules. This means that they override the existing forwarding
> rules.
> +        There are four new rules added for each insertion. Two ingress
> and two
> +        egress, The first ingress rule sends all traffic destined for the
> +        application into the VNF ingress port and the second rule takes
> all
> +        traffic destined to the application from the VNF egress port to
> the
> +        application, the priorities are such that the second rule is
> always
> +        checked first. In the egress direction the rules are similar if
> the
> +        traffic is from the application it is sent to the VNF egress port
> +        and if if is from the application and is from the VNF ingress
> port it
> +        is delivered to the destination. Additional VNFs can be chained
> +        together to create a sequence of operations.
> +
> +      </li>
> +
> +      <li>
> +        One priority-0 fallback flow that matches all packets and
> advances to
> +        the next table.
> +      </li>
> +    </ul>
> +
> +    <h3>Ingress Table 8: <code>from-lport</code> QoS marking</h3>
>
>      <p>
>        Logical flows in this table closely reproduce those in the
> @@ -382,7 +428,7 @@
>        </li>
>      </ul>
>
> -    <h3>Ingress Table 8: LB</h3>
> +    <h3>Ingress Table 9: LB</h3>
>
>      <p>
>        It contains a priority-0 flow that simply moves traffic to the next
> @@ -395,7 +441,7 @@
>        connection.)
>      </p>
>
> -    <h3>Ingress Table 9: Stateful</h3>
> +    <h3>Ingress Table 10: Stateful</h3>
>
>      <ul>
>        <li>
> @@ -432,7 +478,7 @@
>        </li>
>      </ul>
>
> -    <h3>Ingress Table 10: ARP/ND responder</h3>
> +    <h3>Ingress Table 11: ARP/ND responder</h3>
>
>      <p>
>        This table implements ARP/ND responder in a logical switch for known
> @@ -582,7 +628,7 @@ nd_na {
>        </li>
>      </ul>
>
> -    <h3>Ingress Table 11: DHCP option processing</h3>
> +    <h3>Ingress Table 12: DHCP option processing</h3>
>
>      <p>
>        This table adds the DHCPv4 options to a DHCPv4 packet from the
> @@ -642,7 +688,7 @@ next;
>        </li>
>      </ul>
>
> -    <h3>Ingress Table 12: DHCP responses</h3>
> +    <h3>Ingress Table 13: DHCP responses</h3>
>
>      <p>
>        This table implements DHCP responder for the DHCP replies generated
> by
> @@ -724,7 +770,7 @@ output;
>        </li>
>      </ul>
>
> -    <h3>Ingress Table 13 Destination Lookup</h3>
> +    <h3>Ingress Table 14 Destination Lookup</h3>
>
>      <p>
>        This table implements switching behavior.  It contains these logical
> diff --git ovn/northd/ovn-northd.c ovn/northd/ovn-northd.c
> index cc9b934..e85fa4d 100644
> --- ovn/northd/ovn-northd.c
> +++ ovn/northd/ovn-northd.c
> @@ -106,13 +106,14 @@ enum ovn_stage {
>      PIPELINE_STAGE(SWITCH, IN,  PRE_LB,         4, "ls_in_pre_lb")
> \
>      PIPELINE_STAGE(SWITCH, IN,  PRE_STATEFUL,   5, "ls_in_pre_stateful")
> \
>      PIPELINE_STAGE(SWITCH, IN,  ACL,            6, "ls_in_acl")
>  \
> -    PIPELINE_STAGE(SWITCH, IN,  QOS_MARK,       7, "ls_in_qos_mark")
> \
> -    PIPELINE_STAGE(SWITCH, IN,  LB,             8, "ls_in_lb")
> \
> -    PIPELINE_STAGE(SWITCH, IN,  STATEFUL,       9, "ls_in_stateful")
> \
> -    PIPELINE_STAGE(SWITCH, IN,  ARP_ND_RSP,    10, "ls_in_arp_rsp")
>  \
> -    PIPELINE_STAGE(SWITCH, IN,  DHCP_OPTIONS,  11, "ls_in_dhcp_options")
> \
> -    PIPELINE_STAGE(SWITCH, IN,  DHCP_RESPONSE, 12, "ls_in_dhcp_response")
> \
> -    PIPELINE_STAGE(SWITCH, IN,  L2_LKUP,       13, "ls_in_l2_lkup")
>  \
> +    PIPELINE_STAGE(SWITCH, IN,  CHAIN,          7, "ls_in_chain")        \
> +    PIPELINE_STAGE(SWITCH, IN,  QOS_MARK,       8, "ls_in_qos_mark")    \
> +    PIPELINE_STAGE(SWITCH, IN,  LB,             9, "ls_in_lb")
> \
> +    PIPELINE_STAGE(SWITCH, IN,  STATEFUL,      10, "ls_in_stateful")
> \
> +    PIPELINE_STAGE(SWITCH, IN,  ARP_ND_RSP,    11, "ls_in_arp_rsp")
>  \
> +    PIPELINE_STAGE(SWITCH, IN,  DHCP_OPTIONS,  12, "ls_in_dhcp_options")
> \
> +    PIPELINE_STAGE(SWITCH, IN,  DHCP_RESPONSE, 13, "ls_in_dhcp_response")
> \
> +    PIPELINE_STAGE(SWITCH, IN,  L2_LKUP,       14, "ls_in_l2_lkup")
>  \
>                                                                        \
>      /* Logical switch egress stages. */                               \
>      PIPELINE_STAGE(SWITCH, OUT, PRE_LB,       0, "ls_out_pre_lb")     \
> @@ -2872,6 +2873,260 @@ build_acls(struct ovn_datapath *od, struct hmap
> *lflows)
>      }
>  }
>
> +static int
> +cmp_port_pair_groups(const void *ppg1_, const void *ppg2_)
> +{
> +    const struct nbrec_logical_port_pair_group *const *ppg1p = ppg1_;
> +    const struct nbrec_logical_port_pair_group *const *ppg2p = ppg2_;
> +    const struct nbrec_logical_port_pair_group *ppg1 = *ppg1p;
> +    const struct nbrec_logical_port_pair_group *ppg2 = *ppg2p;
> +
> +    if (ppg1->n_sortkey == 0 || ppg2->n_sortkey == 0) {
> +        return 0;
> +    }
> +
> +    if (ppg1->n_sortkey == 0) {
> +        return ppg2->n_sortkey == 0 ? -1 : 0;
> +    } else if (ppg2->n_sortkey == 0) {
> +        return 1;
> +    }
> +
> +    const int64_t key1 = ppg1->sortkey[0];
> +    const int64_t key2 = ppg2->sortkey[0];
> +
> +    if (key1 < key2) {
> +        return -1;
> +    } else if (key1 > key2) {
> +        return 1;
> +    } else {
> +        return 0;
> +    }
> +}
> +
> +static void
> +build_chain(struct ovn_datapath *od, struct hmap *lflows, struct hmap
> *ports)
> +{
> +   /*
> +    * TODO Items
> +    *     * IPV6 support
> +    *     * Load balancing support
> +    *     * Bidirectional parameter support
> +    *     * Support modes of VNF (BitW, L2, L3)
> +    *     * Remove port-security on VNF Ports (if set by CMS)
> +    *     * Add some state to allow match that does not require 'inport'
> +    *     * Support visiting the same VNF more than once
> +    *     * Unit tests!
> +    */
> +    const uint16_t ingress_inner_priority = 150;
> +    const uint16_t ingress_outer_priority = 100;
> +    const uint16_t egress_inner_priority = 150;
> +    const uint16_t egress_outer_priority = 100;
> +
> +    struct ovn_port **input_port_array = NULL;
> +    struct ovn_port **output_port_array = NULL;
> +
> +    struct ovn_port *dst_port = NULL;
> +    struct ovn_port *src_port = NULL;
> +
> +    struct nbrec_logical_port_chain *lpc;
> +    struct nbrec_logical_port_pair_group *lppg;
> +    struct nbrec_logical_port_pair *lpp;
> +    struct nbrec_logical_port_chain_classifier *lcc;
> +
> +    char *lcc_match = NULL;
> +    char *lcc_action = NULL;
> +    struct ovn_port *traffic_port;
> +    unsigned int chain_direction = 2;
> +    unsigned int chain_path = 2;
> +    char * chain_match = NULL;
> +
> +    /* Ingress table ls_in_chain: default to passing through to the next
> table
> +     * (priority 0)
> +     */
> +    ovn_lflow_add(lflows, od, S_SWITCH_IN_CHAIN, 0, "1", "next;");
> +
> +    /* No port chains defined therefore, no further work to do here. */
> +    if (!od->nbs->port_chain_classifiers) {
> +        return;
> +    }
> +    /* Iterate through all the port-chains defined for this datapath. */
> +    for (size_t i = 0; i < od->nbs->n_port_chain_classifiers; i++) {
> +
> +        lcc = od->nbs->port_chain_classifiers[i];
> +        /* Get the parameters from the classifier */
> +        lpc = lcc->chain;
> +        //traffic_port = lcc->port;
> +        traffic_port =  ovn_port_find(ports,lcc->port->name);
> +        /* TODO Check port exists. */
> +        struct eth_addr traffic_logical_port_ea;
> +        ovs_be32 traffic_logical_port_ip;
> +        ovs_scan(traffic_port->nbsp->addresses[0],
> +                    ETH_ADDR_SCAN_FMT" "IP_SCAN_FMT,
> +                    ETH_ADDR_SCAN_ARGS(traffic_logical_port_ea),
> +                    IP_SCAN_ARGS(&traffic_logical_port_ip));
>

This is only taking the first IP address in the first LSP address.
OVN allows for multiple LSP addresses, and allows for multiple IP addresses
in each LSP address.


> +        /* Set the port to use as source or destination. */
> +        if (strcmp(lcc->direction,"entry-lport")==0) {
> +            chain_path = 0;
> +        } else {
> +            chain_path = 1;
> +        }
> +        /* Set the direction of the port-chain. */
> +        if (strcmp(lcc->path,"uni-directional") == 0) {
> +            chain_direction = 0;
> +        } else {
> +            chain_direction = 1;
> +        }
> +        /* Set the match parameters. */
> +        chain_match = lcc->match;
> +        /*
> +         * Allocate space for port-pairs + 1. The Extra 1 represents the
> +         * final hop to reach desired destination.
> +         * TODO: We are going to allocate enough space to hold all the
> hops:
> +         *  1 x portGroups + 1. This needs
> +         *  to enhanced to: SUM(port pairs of each port group) + 1
> +         */
> +        input_port_array = xmalloc(sizeof *src_port *
> +                                   lpc->n_port_pair_groups + 1);
> +        output_port_array = xmalloc(sizeof *dst_port *
> +                                  (lpc->n_port_pair_groups + 1));
> +
> +        /* Copy port groups from chain and sort them according to
> sortkey.*/
> +        struct nbrec_logical_port_pair_group **port_pair_groups =
> +                                 xmemdup(lpc->port_pair_groups,
> +                          sizeof *port_pair_groups *
> lpc->n_port_pair_groups);
> +        qsort(*port_pair_groups, lpc->n_port_pair_groups,
> +              sizeof *port_pair_groups, cmp_port_pair_groups);
> +
> +        /* For each port-pair-group in a port chain pull out the
> port-pairs.*/
> +        for (size_t j = 0; j < lpc->n_port_pair_groups; j++) {
> +            lppg = port_pair_groups[j];
> +            for (size_t k = 0; k < lppg->n_port_pairs; k++) {
> +                 /* TODO: Need to add load balancing logic when LB becomes
> +                 * available. Until LB is available just take the first
> +                 * PP in the PPG. */
> +                if (k > 0) {
> +                    static struct vlog_rate_limit rl =
> +                        VLOG_RATE_LIMIT_INIT(1, 1);
> +                    VLOG_WARN_RL(&rl,
> +                        "Currently lacking support for more than \
> +                            one port-pair %"PRIuSIZE"\n",
> +                              lppg->n_port_pairs);
> +                    break;
> +                }
> +                lpp = lppg->port_pairs[k];
> +
> +                input_port_array[j] = lpp->inport ? ovn_port_find(ports,
> +                                       lpp->inport->name) : NULL;
> +                output_port_array[j] = lpp->outport ? ovn_port_find(ports,
> +                                        lpp->outport->name) : NULL;
> +            }
> +        /* At this point we need to determine the final hop port to add to
> +         * the chain. This defines the complete path for packets through
> +         * the chain. */
> +
> +        /*
> +         * Insert the lowest priorty rule dest is src-logical-port
> +         */
>

The change in indentation here is a bit confusing.


> +    /* TODO add LCC match to match */
> +    if (chain_path == 0) { /* Path starting from entry port */
> +        lcc_match =  xasprintf("ip4.src == "IP_FMT,
> +                                IP_ARGS(traffic_logical_port_ip));
> +        lcc_action = xasprintf("outport = %s; output;",
> +                                input_port_array[0]->json_key);
> +    } else {
> +        lcc_match =  xasprintf("ip4.dst == "IP_FMT,
> +                                IP_ARGS(traffic_logical_port_ip));
> +        lcc_action = xasprintf("outport = %s; output;",
> +                                input_port_array[0]->json_key);
> +    }
> +    ovn_lflow_add(lflows, od, S_SWITCH_IN_CHAIN, ingress_outer_priority,
> +                       lcc_match, lcc_action);
> +    free(lcc_match);
> +    free(lcc_action);
> +    for (size_t j = 0; j < lpc->n_port_pair_groups; j++) {
> +
> +        /* Completed first catch all rule for this port-chain. */
> +
> +        /* Apply inner rules flows */
> +        if (chain_path == 0) { /* Path starting from entry port */
> +            lcc_match = xasprintf("ip4.src == "IP_FMT" && inport == %s",
> +                                IP_ARGS(traffic_logical_port_ip),
> +                                output_port_array[j]->json_key);
> +        } else { /* Path starting from destination port. */
> +            lcc_match = xasprintf("ip4.dst == "IP_FMT" && inport == %s",
> +                                IP_ARGS(traffic_logical_port_ip),
> +                                output_port_array[j]->json_key);
> +
> +        }
> +        if (j == (lpc->n_port_pair_groups-1)) {
> +             lcc_action = xasprintf("next;");
> +
> +        } else {
> +            lcc_action = xasprintf("outport = %s; output;",
> +                                    input_port_array[j+1]->json_key);
> +        }
> +        ovn_lflow_add(lflows, od, S_SWITCH_IN_CHAIN,
> ingress_inner_priority,
> +                        lcc_match, lcc_action);
> +        free(lcc_match);
> +        free(lcc_action);
> +    }
> +    }
> +    if (chain_direction == 1) { /* bi-directional chain */
> +        /*
> +         * Insert the lowest priorty rule dest is src-logical-port
> +         */
> +        /* TODO add LCC match to match */
> +        if (chain_path == 0) { /* Path from ource port. */
> +            lcc_match =  xasprintf("ip4.dst == "IP_FMT,
> +                                    IP_ARGS(traffic_logical_port_ip));
> +            lcc_action = xasprintf("outport = %s; output;",
> +                                    output_port_array[0]->json_key);
> +        } else { /* Path from destination port. */
> +            lcc_match =  xasprintf("ip4.src == "IP_FMT,
> +                                    IP_ARGS(traffic_logical_port_ip));
> +            lcc_action = xasprintf("outport = %s; output;",
> +                                    output_port_array[0]->json_key);
> +        }
> +        ovn_lflow_add(lflows, od, S_SWITCH_IN_CHAIN,
> egress_outer_priority,
> +                       lcc_match, lcc_action);
> +        free(lcc_match);
> +        free(lcc_action);
> +        for (size_t j = 0; j < lpc->n_port_pair_groups; j++) {
> +
> +            /* Completed first catch all rule for this port-chain. */
> +
> +            /* Apply inner rules flows */
> +            if (chain_direction == 0) { /* Path from source port. */
> +                lcc_match = xasprintf("ip4.dst == "IP_FMT" && inport ==
> %s",
> +                                       IP_ARGS(traffic_logical_port_ip),
> +                                       input_port_array[j]->json_key);
> +
> +            } else { /* Path from destination port. */
> +                lcc_match = xasprintf("ip4.src == "IP_FMT" && inport ==
> %s",
> +                                IP_ARGS(traffic_logical_port_ip),
> +                                input_port_array[j]->json_key);
> +
> +            }
> +             if (j == (lpc->n_port_pair_groups-1)) {
> +             lcc_action = xasprintf("next;");
> +
> +        } else {
> +             lcc_action = xasprintf("outport = %s; output;",
> +                                     output_port_array[j]->json_key);
> +
> +        }
> +            ovn_lflow_add(lflows, od, S_SWITCH_IN_CHAIN,
> +                          egress_inner_priority, lcc_match, lcc_action);
> +            free(lcc_match);
> +            free(lcc_action);
> +        }
> +    }
> +
> +    free(input_port_array);
> +    free(output_port_array);
> +    free(port_pair_groups);
> +    }
> +}
>  static void
>  build_qos(struct ovn_datapath *od, struct hmap *lflows) {
>      ovn_lflow_add(lflows, od, S_SWITCH_IN_QOS_MARK, 0, "1", "next;");
> @@ -3000,7 +3255,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>      struct ds actions = DS_EMPTY_INITIALIZER;
>
>      /* Build pre-ACL and ACL tables for both ingress and egress.
> -     * Ingress tables 3 through 9.  Egress tables 0 through 6. */
> +     * Ingress tables 3 through 10.  Egress tables 0 through 6. */
>      struct ovn_datapath *od;
>      HMAP_FOR_EACH (od, key_node, datapaths) {
>          if (!od->nbs) {
> @@ -3011,6 +3266,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          build_pre_lb(od, lflows);
>          build_pre_stateful(od, lflows);
>          build_acls(od, lflows);
> +        build_chain(od, lflows, ports);
>          build_qos(od, lflows);
>          build_lb(od, lflows);
>          build_stateful(od, lflows);
> @@ -3083,9 +3339,9 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          ovn_lflow_add(lflows, od, S_SWITCH_IN_PORT_SEC_IP, 0, "1",
> "next;");
>      }
>
> -    /* Ingress table 10: ARP/ND responder, skip requests coming from
> localnet
> -     * and vtep ports. (priority 100); see ovn-northd.8.xml for the
> -     * rationale. */
> +    /* Ingress table 11: ARP/ND responder, skip requests coming from
> localnet
> +     * ports. (priority 100). */
> +
>      HMAP_FOR_EACH (op, key_node, ports) {
>          if (!op->nbsp) {
>              continue;
> @@ -3100,7 +3356,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          }
>      }
>
> -    /* Ingress table 10: ARP/ND responder, reply for known IPs.
> +    /* Ingress table 11: ARP/ND responder, reply for known IPs.
>       * (priority 50). */
>      HMAP_FOR_EACH (op, key_node, ports) {
>          if (!op->nbsp) {
> @@ -3193,7 +3449,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          }
>      }
>
> -    /* Ingress table 10: ARP/ND responder, by default goto next.
> +    /* Ingress table 11: ARP/ND responder, by default goto next.
>       * (priority 0)*/
>      HMAP_FOR_EACH (od, key_node, datapaths) {
>          if (!od->nbs) {
> @@ -3203,7 +3459,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          ovn_lflow_add(lflows, od, S_SWITCH_IN_ARP_ND_RSP, 0, "1",
> "next;");
>      }
>
> -    /* Logical switch ingress table 11 and 12: DHCP options and response
> +    /* Logical switch ingress table 12 and 13: DHCP options and response
>           * priority 100 flows. */
>      HMAP_FOR_EACH (op, key_node, ports) {
>          if (!op->nbsp) {
> @@ -3307,7 +3563,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          }
>      }
>
> -    /* Ingress table 11 and 12: DHCP options and response, by default
> goto next.
> +    /* Ingress table 12 and 13: DHCP options and response, by default
> goto next.
>       * (priority 0). */
>
>      HMAP_FOR_EACH (od, key_node, datapaths) {
> @@ -3319,7 +3575,8 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          ovn_lflow_add(lflows, od, S_SWITCH_IN_DHCP_RESPONSE, 0, "1",
> "next;");
>      }
>
> -    /* Ingress table 13: Destination lookup, broadcast and multicast
> handling
> +
> +    /* Ingress table 14: Destination lookup, broadcast and multicast
> handling
>       * (priority 100). */
>      HMAP_FOR_EACH (op, key_node, ports) {
>          if (!op->nbsp) {
> @@ -3339,7 +3596,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>                        "outport = \""MC_FLOOD"\"; output;");
>      }
>
> -    /* Ingress table 13: Destination lookup, unicast handling (priority
> 50), */
> +    /* Ingress table 14: Destination lookup, unicast handling (priority
> 50), */
>      HMAP_FOR_EACH (op, key_node, ports) {
>          if (!op->nbsp) {
>              continue;
> @@ -3439,7 +3696,7 @@ build_lswitch_flows(struct hmap *datapaths, struct
> hmap *ports,
>          }
>      }
>
> -    /* Ingress table 13: Destination lookup for unknown MACs (priority
> 0). */
> +    /* Ingress table 14: Destination lookup for unknown MACs (priority
> 0). */
>      HMAP_FOR_EACH (od, key_node, datapaths) {
>          if (!od->nbs) {
>              continue;
> diff --git ovn/ovn-architecture.7.xml ovn/ovn-architecture.7.xml
> index d8114f1..3d8d50c 100644
> --- ovn/ovn-architecture.7.xml
> +++ ovn/ovn-architecture.7.xml
> @@ -383,6 +383,19 @@
>      </li>
>
>      <li>
> +      <dfn>Logical port chains</dfn> are logical references to virtual
> network
> +      functions (VNF). Adding a logical port chain requires adding one or
> more
> +      VNFs to a port chain and then steering traffic into the port chain
> by
> +      adding the port chain to a port chain classifier. The port chain
> +      classifier defines the flows to direct into the port chain, the
> +      direction to apply to the flows on ( to or from a logical port) and
> +      whether to apply the port chain in one direction (uni-directional)
> or
> +      both directions, (bi-directional). This separation enables the port
> +      chain to be defined once and used in several classifiers.
> +      See <code>Life Cycle of an inserted VNF</code>, below, for details.
> +    </li>
> +
> +    <li>
>        <p>
>          <dfn>Logical ports</dfn> represent the points of connectivity in
> and
>          out of logical switches and logical routers.  Some common types of
> @@ -566,6 +579,84 @@
>      </li>
>    </ol>
>
> + <h2>Life Cycle of an Inserted Virtual Network Function (VNF)</h2>
> +
> + <p>
> +   OVN provides an abstraction to enable the insertion of an arbitrary
> virtual
> +   network function (VNF) into the path of traffic to or from an
> application.
> +   A VNF is different from an application VM in that it acts on traffic
> +   between applications, and in most cases does not terminate a flow.
> Proxy
> +   functions are an exception as they terminate the flow from the source
> and
> +   create a new flow to the destination. Examples of VNFs are security
> +   functions (e.g. intrusion detection systems, firewalls), load
> balancers,
> +   and traffic enhancement services.
> + </p>
> + <p>
> +   The requirements on the VNF to be inserted here are minimal: it must
> act as
> +   a <code>bump in the wire (BITW)</code> and can have one or two virtual
> +   network ports for traffic. If it has two network ports, it accepts
> traffic
> +   on one port and transmits it out the other; if it has only one port,
> that
> +   port serves both purposes. The requirement for the VNF to act as a BITW
> +   removes the need for the VNF to participate in L2/L3 networking, which
> +   provides improved agility and reduces the coupling between OVN and the
> VNF.
> +   Not havign the VNF participate in L2/L3 networking makes scale up/down
> +   scenarios easier to implement.
> + </p>
> + <p>
> +   The steps in this example refer to the details of the OVN Northbound
> +   database schema. There is a new table in the OVN Northbound database to
> +   direct traffic to VNFs called <code>port_chain_classifiers</code>,
> which
> +   contains the required information to direct traffic. This table takes
> as
> +   input a port chain which defines an ordered list of VNFs, and
> +   classification parameters to define the traffic to send to the port
> chain.
> +   The same port chain can be used for multiple applications, as there is
> +   typically an N:1 relationship between applications and VNFs. A single
> +   VNF may be part of several service insertions, but each one is
> logically
> +   separate.
> + </p>
> + <p>
> +   The following steps are an overview to inserting a new VNF into the
> traffic
> +   path. The sections below go into each step in more detail.
> + </p>
> + <ol>
> +   <li>
> +     The CMS administrator creates a new virtual network function
> +     <code>(VNF)</code>, using the CMS user interface or API. The CMS
> +     administrator creates the logical ports (ingress and egress) for the
> VNF.
> +     If the CMS is OpenStack, this creates a reusable port-pair defining
> the
> +     interface to the VNF. The administrator also typically creates a
> separate
> +     management port for the VNF, but that is not relevant to the service
> +     insertion workflow. A single VNF can participate with several
> +     applications, either as a security VM, protecting multiple
> applications,
> +     or as a load balancer VM, distributing load across multiple
> applications.
> +   </li>
> +
> +   <li>
> +    <p>
> +     The CMS administrator attaches the VNF port pair to a port pair
> group.
> +     The purpose o fthe port pair group is to enable load balancing across
> +     multiple port-pairs. The port-pair-group is added to an ordered list
> of
> +     port-pair-groups contained in a port chain. At this point the
> port-chain
> +     is just a logicial set of operations with no flows attached.
> +     </p>
> +     <p> Creating a port-chain-classifier, defining the flow
> classification
> +     parameters and the chain to operate on flows inserts a row into the
> +     <code>port-chain-classifier</code> table in the OVN northbound
> +     database. This directs traffic to go through the VNF chain, applying
> +     the operations defined in the port chain.
> +   </p>
> +   </li>
> +
> +   <li>
> +     <p>
> +       Eventually, the application VM shuts down and the CMS removes the
> +       <code>port-chain-classifier</code>.  (However, it is harmless if
> it
> +       remains, since no traffic will be sent to the port-chain unless it
> is
> +       included in a port-chain-classifier.
> +     </p>
> +  </li>
> +</ol>
> +
>    <h2>Life Cycle of a Container Interface Inside a VM</h2>
>
>    <p>
> diff --git ovn/ovn-nb.ovsschema ovn/ovn-nb.ovsschema
> index dd0ac3d..908ab7a 100644
> --- ovn/ovn-nb.ovsschema
> +++ ovn/ovn-nb.ovsschema
> @@ -1,7 +1,7 @@
>  {
>      "name": "OVN_Northbound",
> -    "version": "5.5.0",
> -    "cksum": "2099428463 14236",
> +    "version": "5.5.1",
> +    "cksum": "2551217980 18715",
>      "tables": {
>          "NB_Global": {
>              "columns": {
> @@ -30,6 +30,24 @@
>                                             "refType": "strong"},
>                                     "min": 0,
>                                     "max": "unlimited"}},
> +                "port_chain_classifiers":
> +                              {"type":
> +                                 {"key": {"type": "uuid",
> +                                  "refTable": "Logical_Port_Chain_
> Classifier",
> +                                  "refType": "strong"},
> +                                  "min": 0,
> +                                  "max": "unlimited"}},
> +                "port_chains":
> +                              {"type": {"key": {"type": "uuid",
> +                                        "refTable": "Logical_Port_Chain",
> +                                        "refType": "strong"},
> +                                        "min": 0,
> +                                        "max": "unlimited"}},
>

Why is the port_chains column in the Logical_Switch table necessary?
The ovn-northd code only uses Logical_Port_Chain from references in the
Logical_Port_Chain_Classifier table.

+                "port_pairs":  {"type": {"key": {"type": "uuid",
> +                                         "refTable": "Logical_Port_Pair",
> +                                         "refType": "strong"},
> +                                         "min": 0,
> +                                         "max": "unlimited"}},
>

Why is the port_pairs column in the Logical_Switch table necessary?
The ovn-northd code only uses Logical_Port_Pairs from references in the
Logical_Port_Pair_Group table.


>                  "acls": {"type": {"key": {"type": "uuid",
>                                            "refTable": "ACL",
>                                            "refType": "strong"},
> @@ -98,6 +116,71 @@
>                               "min": 0, "max": "unlimited"}}},
>              "indexes": [["name"]],
>              "isRoot": false},
> +        "Logical_Port_Chain": {
> +            "columns": {
> +                "name": {"type": "string"},
> +                "port_pair_groups":  {"type":
> +                                        {"key": {"type": "uuid",
> +                                         "refTable":
> "Logical_Port_Pair_Group",
> +                                         "refType": "strong"},
> +                                         "min": 0, "max": "unlimited"}},
> +                "external_ids": {
> +                    "type": {"key": "string", "value": "string",
> +                             "min": 0, "max": "unlimited"}}},
> +            "isRoot": false},
> +        "Logical_Port_Pair_Group": {
> +            "columns": {
> +                "name": {"type": "string"},
> +                "port_pairs":  {"type":
> +                                  {"key": {"type": "uuid",
> +                                           "refTable":
> "Logical_Port_Pair",
> +                                           "refType": "strong"},
> +                                           "min": 0, "max": "unlimited"}},
> +                "external_ids": {
> +                    "type": {"key": "string", "value": "string",
> +                             "min": 0, "max": "unlimited"}},
> +                "sortkey": {"type": {"key": {"type": "integer"},
> +                                     "min": 0, "max": 1}}
> +            },
> +            "isRoot": false},
> +        "Logical_Port_Chain_Classifier": {
> +            "columns": {
> +                "name": {"type": "string"},
> +                "chain": {"type": {"key": {"type": "uuid",
> +                                             "refTable":
> "Logical_Port_Chain",
> +                                             "refType": "strong"},
> +                                     "min": 0, "max": 1}},
> +                "port": {"type": {"key": {"type": "uuid",
> +                                             "refTable":
> "Logical_Switch_Port",
> +                                             "refType": "strong"},
> +                                     "min": 0, "max": 1}},
> +                "direction": {"type":
> +                           {"key": {"type": "string",
> +                            "enum": ["set", ["entry-lport",
> "exit-lport"]]}}},
> +                "path": {"type":
> +                           {"key": {"type": "string",
> +                            "enum": ["set",
> +                                    ["uni-directional",
> "bi-directional"]]}}},
>

IMO the word "path" does not really fit the contents of "uni-directional",
"bi-directional".


> +                "match": {"type": "string"},
> +                "external_ids": {
> +                    "type": {"key": "string", "value": "string",
> +                             "min": 0, "max": "unlimited"}}},
> +            "isRoot": false},
> +        "Logical_Port_Pair": {
> +            "columns": {
> +                "name": {"type": "string"},
> +                "outport": {"type": {"key": {"type": "uuid",
> +                                             "refTable":
> "Logical_Switch_Port",
> +                                             "refType": "strong"},
> +                                     "min": 0, "max": 1}},
> +                "inport": {"type": {"key": {"type": "uuid",
> +                                            "refTable":
> "Logical_Switch_Port",
> +                                            "refType": "strong"},
> +                                    "min": 0, "max": 1}},
> +                "external_ids": {
> +                    "type": {"key": "string", "value": "string",
> +                             "min": 0, "max": "unlimited"}}},
> +            "isRoot": false},
>          "Address_Set": {
>              "columns": {
>                  "name": {"type": "string"},
> diff --git ovn/ovn-nb.xml ovn/ovn-nb.xml
> index 88a6082..16ee066 100644
> --- ovn/ovn-nb.xml
> +++ ovn/ovn-nb.xml
> @@ -121,6 +121,35 @@
>        </p>
>      </column>
>
> +    <column name="port_chain_classifiers">
> +      <p>
> +        The logical port chain classifiers defining the path of traffic
> +        to take through attached port chains. The port chain classifier
> +        defines the logical port traffic is to or from and whether the
> +        chain is applied to flows in one or both directions.
> +      </p>
> +    </column>
> +
> +    <column name="port_chains">
> +      <p>
> +        The logical port-chains connected to the logical switch. Port
> chains
> +        can be reused across multiple classifiers.
> +      </p>
> +      <p>
> +        It is an error for port-pairs within a port chain to span multiple
> +        logical switches.
> +      </p>
> +    </column>
> +
> +    <column name="port_pairs">
> +      <p>
> +        The logical chains that define the service path.
> +      </p>
> +      <p>
> +        Logical port pairs cannot currently cross logical switch
> boundaries.
> +      </p>
> +    </column>
> +
>      <column name="load_balancer">
>        Load balance a virtual ipv4 address to a set of logical port
> endpoint
>        ipv4 addresses.
> @@ -156,6 +185,157 @@
>      </group>
>    </table>
>
> +  <table name="Logical_Port_Chain_Classifier" title="Logical port chain
> classifier">
> +    <p>
> +      Each row represents one logical port chain classifier
> +    </p>
> +    <column name="name">
> +      <p>
> +        A name for the logical chain classifer. This name has no special
> +        meaning or purpose other than to provide convenience for human
> +        interaction with the ovn-nb database.  There is no requirement for
> +        the name to be unique.  The logical chain classifier's UUID should
> +        be used as the unique identifier.
> +      </p>
> +    </column>
> +    <column name="port">
> +      <p>
> +        The logical port that is either the src or dst of flows for the
> +        port chain.
> +      </p>
> +      <p>
> +        It is an error for this to be NULL.
> +      </p>
> +    </column>
> +    <column name="chain">
> +      <p>
> +        The port chain for the flows to traverse. The port chain can be
> +        reused in multiple classifiers.
> +      </p>
> +    </column>
> +    <column name="direction">
> +      <p>
> +        The direction of the flows through the port chain, this can be
> either
> +        "uni-directional" or "bi-directional".
> +      </p>
> +    </column>
> +    <column name="path">
> +      <p>
> +       The path of the flow from or to the logical port. The valid values
> are
> +       "entry-lport" or "exit-lport". If the path is "entry-lport" the
> rules
> +       are applied to traffic leaving the entry-lport, if the path is
> +       "exit-lport" the port-chain is applied to traffic going to the
> lport.
> +      </p>
> +    </column>
> +    <column name="match">
> +      <p>
> +       The match is an optional match statement that will filter flows
> going
> +       to the chain as defined by any valid openvswitch matches.
> +      </p>
> +      <p>
> +        Care should be taken using match statements to ensure they do not
> +        conflicit with in logical port the is the src or dst of flows. In
> +        addition when operating on "bi-directional" flows care shoudl be
> taken
> +        to only use match statements that work in both directions.
> +      </p>
> +    </column>
> +    <group title="Common Columns">
> +      <column name="external_ids">
> +        See <em>External IDs</em> at the beginning of this document.
> +      </column>
> +    </group>
> +  </table>
> +    <table name="Logical_Port_Chain" title="Logical port chain">
> +    <p>
> +      Each row represents one logical port chain
> +    </p>
> +
> +    <column name="name">
> +      <p>
> +        A name for the logical chain.  This name has no special meaning or
> +        purpose other than to provide convenience for human interaction
> with
> +        the ovn-nb database.  There is no requirement for the name to be
> +        unique. The logical chains's UUID should be used as the unique
> +        identifier.
> +      </p>
> +    </column>
> +
> +    <column name="port_pair_groups">
> +      <p>
> +        The logical list of port pairs that the flow goes through.
> +      </p>
> +
> +      <p>
> +        It is an error for a port pair group to be empty.
> +      </p>
> +    </column>
> +
> +    <group title="Common Columns">
> +      <column name="external_ids">
> +        See <em>External IDs</em> at the beginning of this document.
> +      </column>
> +    </group>
> +  </table>
> +  <table name="Logical_Port_Pair_Group" title="logical port pair groups">
> +    <p>
> +      An ordered port pair list
> +    </p>
> +
> +    <column name="name">
> +      <p>
> +        Logical port pair group name
> +      </p>
> +    </column>
> +
> +    <column name="port_pairs">
> +      <p>
> +        port pair for this group
> +      </p>
> +    </column>
> +
> +    <column name="sortkey">
> +      <p>
> +        An integer used for ordering instances of
> +        <ref table="Logical_Port_Pair_Group"/> in the
> +        <ref column="port_pairs" table="Logical_Port_Chain"/> column
> +        of <ref table="Logical_Port_Chain"/>.
> +      </p>
> +    </column>
> +    <group title="Common Columns">
> +      <column name="external_ids">
> +        See <em>External IDs</em> at the beginning of this document.
> +      </column>
> +    </group>
> +  </table>
> +
> +  <table name="Logical_Port_Pair" title="logical port pairs">
> +    <p>
> +      Ports pairs defining the service
> +    </p>
> +
> +    <column name="name">
> +      <p>
> +        Logical port pair
> +      </p>
> +    </column>
> +
> +    <column name="outport">
> +      <p>
> +        Out logical port for this port pair. Can be the same value as
> inport.
> +      </p>
> +    </column>
> +
> +    <column name="inport">
> +      <p>
> +        In logical port for this port pair.
> +      </p>
> +    </column>
> +    <group title="Common Columns">
> +      <column name="external_ids">
> +        See <em>External IDs</em> at the beginning of this document.
> +      </column>
> +    </group>
> +  </table>
>    <table name="Logical_Switch_Port" title="L2 logical switch port">
>      <p>
>        A port within an L2 logical switch.
> @@ -912,6 +1092,13 @@
>            ICMP unreachable message for other IP-based protocols.
>            <code>Not implemented--currently treated as drop</code>
>          </li>
> +
> +        <li>
> +          <code>sfc</code>: Forward the packet into a logical port chain.
> +          The chain to be used -- as well as any other attributes that
> +          determine the behavior of the packet while in the chain -- are
> +          provided via <ref column="options"/>.
> +        </li>
>        </ul>
>      </column>
>

The above code snippet seems to be left over from the previous revision.


>
> diff --git ovn/utilities/ovn-nbctl.8.xml ovn/utilities/ovn-nbctl.8.xml
> index 1913488..91a159e 100644
> --- ovn/utilities/ovn-nbctl.8.xml
> +++ ovn/utilities/ovn-nbctl.8.xml
> @@ -288,6 +288,224 @@
>
>      </dl>
>
> +    <h1>Logical Port Chain Classifier Commands</h1>
> +
> +    <dl>
> +      <dt><code>lsp-chain-classifier-add</code></dt>
> +      <dd>
> +        <p>
> +          Creates a new, logical port chain classifier.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--may-exist</code> | <code>--add-duplicate</code>]
> <code>lsp-chain-classifier-add</code> <var>switch</var>
> <var>port-chain</var> <var>port</var> <var>direction</var> <var>path</var>
> <var>match</var> [<var>classifier</var>] </dt>
> +      <dd>
> +        <p>
> +          Creates a new logical port-chain-classifier named
> +          <var>classifier</var>. This operation inserts the chaining rules
> +          into the ovn database.
> +        </p>
> +
> +        <p>
> +          The OVN northbound database schema does not require logical port
> +          chain classifier names to be unique, but the whole point to the
> +          names is to provide an easy way for humans to refer to the port
> +          chain classifiers, making duplicate names unhelpful. Thus,
> without
> +          any options, this command regards it as an error if
> +          <var>clasisfier</var> is a duplicate name.  With
> +          <code>--may-exist</code>, adding a duplicate name succeeds but
> does
> +          not create a new logical port-chain. With
> +          <code>--add-duplicate</code>, the command really creates a new
> +          logical port-chain-classifier with a duplicate name.  It is an
> error
> +          to specify both options.  If there are multiple logical
> +          port-chainclassifiers with a duplicate name, configure the
> logical
> +          port-chain-classifiers using the UUID instead of the
> +          <var>classifier</var> name.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--if-exists</code>] <code>lsp-chain-classififer-del</code>
> <var>port-chain-classifier</var></dt>
> +      <dd>
> +        Deletes <var>port-chain</var>. It is an error if
> +        <var>port-chain-classifier</var> does not exist, unless
> +        <code>--if-exists</code> is specified.
> +      </dd>
> +
> +      <dt><code>lsp-chain-classifier-list</code> <var>switch</var>
> <var>port-chain-classifier</var></dt>
> +      <dd>
> +        Lists all existing port-chain-classifiers on standard output, one
> per
> +        line.
> +      </dd>
> +       <dt><code>lsp-chain-classifier-show</code>
> [<var>port-chain-classifier</var>]</dt>
> +      <dd>
> +        Show all existing port-chain-classifiers on standard output, one
> per
> +        line.
> +      </dd>
> +    </dl>
> +
> +    <h1>Logical Port Chain Commands</h1>
> +
> +    <dl>
> +      <dt><code>lsp-chain-add</code></dt>
> +      <dd>
> +        <p>
> +          Creates a new, logical port chain.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--may-exist</code> | <code>--add-duplicate</code>]
> <code>lsp-chain-add</code> <var>port-chain</var></dt>
> +      <dd>
> +        <p>
> +          Creates a new logical port-chain named <var>port-chain</var>,
> which
> +          initially has no port pair groups. Port chains are made from an
> +          ordered list of port-pair-groups. Each port-pair-group has one
> or
> +          more port-pairs. Every port-pair within a port-pair-group
> should be
> +          of the same type of VNF, but OVN cannot enforce this
> requirement.
> +        </p>
> +
> +        <p>
> +          The OVN northbound database schema does not require logical port
> +          chain names to be unique, but the whole point to the names is to
> +          provide an easy way for humans to refer to the port chains,
> making
> +          duplicate names unhelpful.  Thus, without any options, this
> command
> +          regards it as an error if <var>port-chain</var> is a duplicate
> name.
> +          With <code>--may-exist</code>, adding a duplicate name succeeds
> but
> +          does not create a new logical port-chain. With
> +          <code>--add-duplicate</code>, the command really creates a new
> +          logical port-chain with a duplicate name.  It is an error to
> specify
> +          both options. If there are multiple logical port-chains with a
> +          duplicate name, configure the logical port-chains using the UUID
> +          instead of the <var>port-chain</var> name.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--if-exists</code>] <code>lsp-chain-del</code>
> <var>port-chain</var></dt>
> +      <dd>
> +        Deletes <var>port-chain</var>.  It is an error if
> +        <var>port-chain</var> does not exist, unless
> <code>--if-exists</code>
> +        is specified.
> +      </dd>
> +
> +      <dt><code>lsp-chain-list</code> <var>switch</var>
> <var>port-chain</var></dt>
> +      <dd>
> +        Lists all existing port-chains on standard output, one per line.
> +      </dd>
> +       <dt><code>lsp-chain-show</code> [<var>port-chain</var>]</dt>
> +      <dd>
> +        Show all existing port-chains on standard output, one per line.
> +      </dd>
> +    </dl>
> +
> +    <h1>Logical Port Pair Group Commands</h1>
> +
> +    <dl>
> +      <dt><code>lsp-pair-group-add</code></dt>
> +      <dd>
> +        <p>
> +          Creates a new, logical port pair group.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--may-exist</code> | <code>--add-duplicate</code>]
> <code>lsp-pair-group-add</code> <var>port-pair-group</var></dt>
>

This does not match the syntax defined below with CHAIN and [OFFSET].

Mickey



> +      <dd>
> +        <p>
> +          Creates a new logical port pair group named
> +          <var>port-pair-group</var>, which initially has no port pairs.
> Port
> +          pair groups are made from a collection of port-pairs. Each
> +          port-pair-group has one or more port-pairs. Every port-pair
> within a
> +          port-pair-group should be of the same type of VNF, but OVN
> cannot
> +          enforce this requirement.
> +        </p>
> +
> +        <p>
> +          The OVN northbound database schema does not require logical
> port pair
> +          group names to be unique, but the whole point to the names is to
> +          provide an easy way for humans to refer to the port pair groups,
> +          making duplicate names unhelpful.  Thus, without any options,
> this
> +          command regards it as an error if <var>port-pair-group</var> is
> a
> +          duplicate name. With <code>--may-exist</code>, adding a
> duplicate
> +          name succeeds but does not create a new logical port-pair-group.
> +          With <code>--add-duplicate</code>, the command really creates a
> new
> +          logical port-pair-group with a duplicate name. It is an error to
> +          specify both options.  If there are multiple logical port pair
> +          groups with a duplicate name, configure the logical port pair
> groups
> +          using the UUID instead of the <var>port-pair-group</var> name.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--if-exists</code>] <code>lsp-pair-group-del</code>
> <var>port-pair-group</var></dt>
> +      <dd>
> +        Deletes <var>port-pair-group</var>.  It is an error if
> +        <var>port-pair-group</var> does not exist, unless
> +        <code>--if-exists</code> is specified.
> +      </dd>
> +
> +      <dt><code>lsp-pair-group-list</code> <var>port-chain</var> </dt>
> +      <dd>
> +        <p>
> +        Lists all existing port-pair-groups in a port-chain on standard
> +        output, one per line.
> +        </p>
> +      </dd>
> +      <dt><code>lsp-pair-group-add-port-pair</code> </dt>
> +      <dd>
> +        <p>
> +        Adds a port-pair to an existing port-pair group.
> +        </p>
> +      </dd>
> +      <dt><code>lsp-pair-group-del-port-pair</code>
> <var>port-pair-group</var> <var>port-pair</var> </dt>
> +      <dd>
> +        Deletes an existing port-pair from an existing port-pair group.
> +      </dd>
> +      </dl>
> +
> +      <h1>Logical Port Pair Commands</h1>
> +
> +      <dl>
> +      <dt><code>lsp-pair-add</code></dt>
> +      <dd>
> +        <p>
> +          Creates a new, logical port pair.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--may-exist</code> | <code>--add-duplicate</code>]
> <code>lsp-pair-add</code> <var>port-pair</var></dt>
> +      <dd>
> +        <p>
> +          Creates a new logical port pair named <var>port-pair</var>. Port
> +          pairs are made from one or two logical switch ports. The ports
> +          require that the attached VNF passes traffic unchanged between
> the
> +          two ports of in the case of a single port in and out the same
> port.
> +        </p>
> +
> +        <p>
> +          The OVN northbound database schema does not require logical port
> +          pair names to be unique, but the whole point to the names is to
> +          provide an easy way for humans to refer to the port pairs,
> making
> +          duplicate names unhelpful. Thus, without any options, this
> command
> +          regards it as an error if <var>port-pair</var> is a duplicate
> name.
> +          With <code>--may-exist</code>, adding a duplicate name succeeds
> but
> +          does not create a new logical port-pair. With
> +          <code>--add-duplicate</code>, the command really creates a new
> +          logical port-pair with a duplicate name.  It is an error to
> specify
> +          both options.  If there are multiple logical port pairs with a
> +          duplicate name, configure the logical port pairs using the UUID
> +          instead of the <var>port-pair</var> name.
> +        </p>
> +      </dd>
> +
> +      <dt>[<code>--if-exists</code>] <code>lsp-pair-del</code>
> <var>port-pair</var></dt>
> +      <dd>
> +        Deletes <var>port-pair</var>.  It is an error if
> <var>port-pair</var>
> +        does not exist, unless <code>--if-exists</code> is specified.
> +      </dd>
> +
> +      <dt><code>lsp-pair-list</code> <var>switch</var>
> <var>port-pair-group</var></dt>
> +      <dd>
> +        Lists all existing port pairs on standard output, one per line.
> +      </dd>
> +    </dl>
> +
>      <h1>Logical Router Commands</h1>
>
>      <dl>
> diff --git ovn/utilities/ovn-nbctl.c ovn/utilities/ovn-nbctl.c
> index 900b088..94c732f 100644
> --- ovn/utilities/ovn-nbctl.c
> +++ ovn/utilities/ovn-nbctl.c
> @@ -366,6 +366,43 @@ Logical switch port commands:\n\
>                              set dhcpv4 options for PORT\n\
>    lsp-get-dhcpv4-options PORT  get the dhcpv4 options for PORT\n\
>  \n\
> +Logical port chain classifier commands:\n\
> +  lsp-chain-classifier-add SWITCH CHAIN PORT DIRECTION PATH MATCH
> [NAME]\n\
> +                            add a CHAIN to a CLASSIFIER\n\
> +  lsp-chain-classifier-del CHAIN \n\
> +                            remove classifier from CHAIN\n\
> +  lsp-chain-classifier-list SWITCH [CHAIN]\n\
> +                            print classifiers for SWITCH [CHAIN]\n\
> +\n\
> +Logical port chain commands:\n\
> +  lsp-chain-add SWITCH CHAIN         create a logical port-chain\n\
> +                                     named CHAIN\n\
> +  lsp-chain-del CHAIN                delete CHAIN\n\
> +  lsp-chain-list [SWITCH]            print the names of all logical\n\
> +                                     port-chains [on SWITCH]\n\
> +  lsp-chain-show SWITCH [CHAIN]      print details on port-chains\n\
> +                                     on SWITCH\n\
> +\n\
> +Logical port pair group commands:\n\
> +  lsp-pair-group-add CHAIN [PAIR-GROUP [OFFSET]]\n\
> +                    create a logical port-pair-group. Optionally,\n\
> +                    indicate the order it should be in chain.\n\
> +  lsp-pair-group-del PAIR-GROUP    delete a port-pair-group, does\n\
> +                                   not delete port-pairs\n\
> +  lsp-pair-group-list CHAIN        print port-pair-groups for a given
> chain\n\
> +  lsp-pair-group-add-port-pair PAIR-GROUP LSP-PAIR add a port pair to a\n\
> +                                                   port-pair-group\n\
> +  lsp-pair-group-del-port-pair PAIR-GROUP LSP-PAIR del a port pair from
> a\n\
> +                                                   port-pair-group\n\
> +\n\
> +Logical port pair commands:\n\
> +  lsp-pair-add SWITCH PORT-IN PORT-OUT [LSP-PAIR]\n\
> +                                     create a logical port-pair\n\
> +  lsp-pair-del LSP-PAIR              delete a port-pair, does\n\
> +                                     not delete ports\n\
> +  lsp-pair-list [SWITCH [LSP-PAIR]]  print the names of all\n\
> +                                     logical port-pairs\n\
> +\n\
>  Logical router commands:\n\
>    lr-add [ROUTER]           create a logical router named ROUTER\n\
>    lr-del ROUTER             delete ROUTER and all its ports\n\
> @@ -764,6 +801,1035 @@ lsp_by_name_or_uuid(struct ctl_context *ctx, const
> char *id,
>
>      return lsp;
>  }
> +
> +/*
> + * Port chain CLI Functions
> + */
> +static const struct nbrec_logical_port_chain *
> +lsp_chain_by_name_or_uuid(struct ctl_context *ctx, const char *id,
> +                           const bool must_exist)
> +{
> +    const struct nbrec_logical_port_chain *lsp_chain = NULL;
> +    bool is_uuid = false;
> +    struct uuid lsp_chain_uuid;
> +
> +    if (uuid_from_string(&lsp_chain_uuid, id)) {
> +        is_uuid = true;
> +        lsp_chain = nbrec_logical_port_chain_get_for_uuid(ctx->idl,
> +
> &lsp_chain_uuid);
> +    }
> +
> +    if (!lsp_chain) {
> +        NBREC_LOGICAL_PORT_CHAIN_FOR_EACH(lsp_chain, ctx->idl) {
> +            if (!strcmp(lsp_chain->name, id)) {
> +                break;
> +            }
> +        }
> +    }
> +    if (!lsp_chain && must_exist) {
> +        ctl_fatal("lsp_chain not found for %s: '%s'",
> +                  is_uuid ? "UUID" : "name", id);
> +    }
> +
> +    return lsp_chain;
> +}
> +static const struct nbrec_logical_port_pair_group *
> +lsp_pair_group_by_name_or_uuid(struct ctl_context *ctx, const char *id,
> +                                const bool must_exist)
> +{
> +    const struct nbrec_logical_port_pair_group *lsp_pair_group = NULL;
> +    bool is_uuid = false;
> +    struct uuid lsp_pair_group_uuid;
> +
> +    if (uuid_from_string(&lsp_pair_group_uuid, id)) {
> +        is_uuid = true;
> +        lsp_pair_group =
> +          nbrec_logical_port_pair_group_get_for_uuid(ctx->idl,
> +
> &lsp_pair_group_uuid);
> +    }
> +
> +    if (!lsp_pair_group) {
> +        NBREC_LOGICAL_PORT_PAIR_GROUP_FOR_EACH(lsp_pair_group, ctx->idl)
> {
> +            if (!strcmp(lsp_pair_group->name, id)) {
> +                break;
> +            }
> +        }
> +    }
> +    if (!lsp_pair_group && must_exist) {
> +        ctl_fatal("lsp_pair_group not found for %s: '%s'",
> +                  is_uuid ? "UUID" : "name", id);
> +    }
> +
> +    return lsp_pair_group;
> +}
> +
> +static const struct nbrec_logical_port_chain_classifier *
> +lsp_chain_classifier_by_name_or_uuid(struct ctl_context *ctx, const char
> *id,
> +                                const bool must_exist)
> +{
> +    const struct nbrec_logical_port_chain_classifier
> +                                *lsp_chain_classifier = NULL;
> +    bool is_uuid = false;
> +    struct uuid lsp_chain_classifier_uuid;
> +
> +    if (uuid_from_string(&lsp_chain_classifier_uuid, id)) {
> +        is_uuid = true;
> +        lsp_chain_classifier =
> +          nbrec_logical_port_chain_classifier_get_for_uuid(ctx->idl,
> +                                              &lsp_chain_classifier_uuid);
> +    }
> +
> +    if (!lsp_chain_classifier) {
> +        NBREC_LOGICAL_PORT_CHAIN_CLASSIFIER_FOR_EACH(lsp_chain_
> classifier,
> +                                                      ctx->idl) {
> +            if (!strcmp(lsp_chain_classifier->name, id)) {
> +                break;
> +            }
> +        }
> +    }
> +    if (!lsp_chain_classifier && must_exist) {
> +        ctl_fatal("lsp_chain_classifier not found for %s: '%s'",
> +                  is_uuid ? "UUID" : "name", id);
> +    }
> +
> +    return lsp_chain_classifier;
> +}
> +
> +static const struct nbrec_logical_port_pair *
> +lsp_pair_by_name_or_uuid(struct ctl_context *ctx, const char *id,
> +                          const bool must_exist)
> +{
> +    const struct nbrec_logical_port_pair *lsp_pair = NULL;
> +    bool is_uuid = false;
> +    struct uuid lsp_pair_uuid;
> +
> +    if (uuid_from_string(&lsp_pair_uuid, id)) {
> +        is_uuid = true;
> +        lsp_pair = nbrec_logical_port_pair_get_for_uuid(ctx->idl,
> +                                                        &lsp_pair_uuid);
> +    }
> +
> +    if (!lsp_pair) {
> +        NBREC_LOGICAL_PORT_PAIR_FOR_EACH(lsp_pair, ctx->idl) {
> +            if (!strcmp(lsp_pair->name, id)) {
> +                break;
> +            }
> +        }
> +    }
> +    if (!lsp_pair && must_exist) {
> +        ctl_fatal("lsp_pair not found for %s: '%s'",
> +                  is_uuid ? "UUID" : "name", id);
> +    }
> +
> +    return lsp_pair;
> +}
> +
> +
> +static void
> +nbctl_lsp_chain_add(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_switch *lswitch;
> +
> +    lswitch = ls_by_name_or_uuid(ctx, ctx->argv[1], true);
> +    const char *lsp_chain_name = ctx->argc > 2 ? ctx->argv[2] : NULL;
> +
> +    const bool may_exist = shash_find(&ctx->options, "--may-exist") !=
> NULL;
> +    const bool add_duplicate = shash_find(&ctx->options,
> +                                 "--add-duplicate") != NULL;
> +    if (may_exist && add_duplicate) {
> +        ctl_fatal("--may-exist and --add-duplicate may not be used
> together");
> +    }
> +
> +    if (lsp_chain_name) {
> +        if (!add_duplicate) {
> +            const struct nbrec_logical_port_chain *lsp_chain;
> +            NBREC_LOGICAL_PORT_CHAIN_FOR_EACH(lsp_chain, ctx->idl) {
> +                if (!strcmp(lsp_chain->name, lsp_chain_name)) {
> +                    if (may_exist) {
> +                        return;
> +                    }
> +                    ctl_fatal("%s: a lsp_chain with this name already
> exists",
> +                              lsp_chain_name);
> +                }
> +            }
> +        }
> +    } else if (may_exist) {
> +        ctl_fatal("--may-exist requires specifying a name");
> +    } else if (add_duplicate) {
> +        ctl_fatal("--add-duplicate requires specifying a name");
> +    }
> +
> +    struct nbrec_logical_port_chain *lsp_chain;
> +    lsp_chain = nbrec_logical_port_chain_insert(ctx->txn);
> +    if (lsp_chain_name) {
> +        nbrec_logical_port_chain_set_name(lsp_chain, lsp_chain_name);
> +    }
> +
> +    /* Insert the logical port-chain into the logical switch. */
> +
> +    nbrec_logical_switch_verify_port_chains(lswitch);
> +    struct nbrec_logical_port_chain  **new_port_chain =
> +              xmalloc(sizeof *new_port_chain * (lswitch->n_port_chains +
> 1));
> +    memcpy(new_port_chain, lswitch->port_chains,
> +              sizeof *new_port_chain * lswitch->n_port_chains);
> +    new_port_chain[lswitch->n_port_chains] =
> +              CONST_CAST(struct nbrec_logical_port_chain *, lsp_chain);
> +    nbrec_logical_switch_set_port_chains(lswitch, new_port_chain,
> +              lswitch->n_port_chains + 1);
> +    free(new_port_chain);
> +}
> +
> +/* Removes lswitch->pair_chain[idx]'. */
> +static void
> +remove_lsp_chain(const struct nbrec_logical_switch *lswitch, size_t idx)
> +{
> +    const struct nbrec_logical_port_chain *lsp_chain =
> +
> lswitch->port_chains[idx];
> +
> +    /* First remove 'lsp-chain' from the array of port-chains.
> +     * This is what will actually cause the logical port-chain to be
> deleted
> +     * when the transaction is sent to the database server
> +     * (due to garbage collection). */
> +    struct nbrec_logical_port_chain **new_port_chain
> +        = xmemdup(lswitch->port_chains,
> +                   sizeof *new_port_chain * lswitch->n_port_chains);
> +    new_port_chain[idx] = new_port_chain[lswitch->n_port_chains - 1];
> +    nbrec_logical_switch_verify_port_chains(lswitch);
> +    nbrec_logical_switch_set_port_chains(lswitch, new_port_chain,
> +                                          lswitch->n_port_chains - 1);
> +    free(new_port_chain);
> +
> +    /* Delete 'lsp-chain' from the IDL.  This won't have a real effect on
> the
> +     * database server (the IDL will suppress it in fact) but it means
> that it
> +     * won't show up when we iterate with
> +     * NBREC_LOGICAL_PORT_CHAIN_FOR_EACH later. */
> +    nbrec_logical_port_chain_delete(lsp_chain);
> +}
> +
> +static void
> +nbctl_lsp_chain_del(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_chain *lsp_chain;
> +    const bool must_exist = !shash_find(&ctx->options, "--if-exists");
> +
> +    lsp_chain = lsp_chain_by_name_or_uuid(ctx, ctx->argv[1], must_exist);
> +    if (!lsp_chain) {
> +        return;
> +    }
> +
> +    /* Find the lswitch that contains 'port-chain', then delete it. */
> +    const struct nbrec_logical_switch *lswitch;
> +    NBREC_LOGICAL_SWITCH_FOR_EACH (lswitch, ctx->idl) {
> +        for (size_t i = 0; i < lswitch->n_port_chains; i++) {
> +            if (lswitch->port_chains[i] == lsp_chain) {
> +                remove_lsp_chain(lswitch,i);
> +                return;
> +            }
> +        }
> +    }
> +}
> +
> +static void
> +print_lsp_chain_entry(struct ctl_context *ctx,
> +                      const struct nbrec_logical_switch *lswitch,
> +                      const char *chain_name_filter,
> +                      const bool show_switch_name)
> +{
> +    struct smap lsp_chains;
> +    size_t i;
> +
> +    smap_init(&lsp_chains);
> +    for (i = 0; i < lswitch->n_port_chains; i++) {
> +        const struct nbrec_logical_port_chain *lsp_chain =
> +
> lswitch->port_chains[i];
> +        if (chain_name_filter && strcmp(chain_name_filter,
> lsp_chain->name)) {
> +            continue;
> +        }
> +        if (show_switch_name) {
> +            smap_add_format(&lsp_chains, lsp_chain->name, UUID_FMT "
> (%s:%s)",
> +                            UUID_ARGS(&lsp_chain->header_.uuid),
> +                            lswitch->name, lsp_chain->name);
> +        } else {
> +            smap_add_format(&lsp_chains, lsp_chain->name, UUID_FMT "
> (%s)",
> +                            UUID_ARGS(&lsp_chain->header_.uuid),
> +                            lsp_chain->name);
> +        }
> +    }
> +
> +    const struct smap_node **nodes = smap_sort(&lsp_chains);
> +    for (i = 0; i < smap_count(&lsp_chains); i++) {
> +        const struct smap_node *node = nodes[i];
> +        ds_put_format(&ctx->output, "%s\n", node->value);
> +    }
> +    smap_destroy(&lsp_chains);
> +    free(nodes);
> +}
> +
> +static void
> +nbctl_lsp_chain_list(struct ctl_context *ctx)
> +{
> +    const char *id = ctx->argc > 1 ? ctx->argv[1] : NULL;
> +    const char *chain_name_filter = ctx->argc > 2 ? ctx->argv[2] : NULL;
> +    const struct nbrec_logical_switch *lswitch;
> +
> +    if (id) {
> +        lswitch = ls_by_name_or_uuid(ctx, id, true);
> +        print_lsp_chain_entry(ctx, lswitch, chain_name_filter, false);
> +    } else {
> +        NBREC_LOGICAL_SWITCH_FOR_EACH(lswitch, ctx->idl) {
> +            if (lswitch->n_port_chains == 0) {
> +                continue;
> +            }
> +            print_lsp_chain_entry(ctx, lswitch, chain_name_filter, true);
> +        }
> +    }
> +}
> +
> +static void
> +print_lsp_chain(const struct nbrec_logical_port_chain *lsp_chain,
> +                  struct ctl_context *ctx)
> +{
> +    ds_put_format(&ctx->output, "lsp-chain "UUID_FMT" (%s)\n",
> +                  UUID_ARGS(&lsp_chain->header_.uuid), lsp_chain->name);
> +    for (size_t i = 0; i < lsp_chain->n_port_pair_groups; i++) {
> +        const struct nbrec_logical_port_pair_group *lsp_pair_group
> +            = lsp_chain->port_pair_groups[i];
> +        ds_put_format(&ctx->output, "    lsp-pair-group %s\n",
> +                       lsp_pair_group->name);
> +        for (size_t j = 0; j < lsp_pair_group->n_port_pairs; j++) {
> +            const struct nbrec_logical_port_pair *lsp_pair =
> +
> lsp_pair_group->port_pairs[j];
> +            ds_put_format(&ctx->output,
> +                           "        lsp-pair %s\n", lsp_pair->name);
> +
> +            const struct nbrec_logical_switch_port *linport =
> lsp_pair->inport;
> +            if (linport) {
> +                ds_put_format(&ctx->output,
> +                           "            lsp-pair inport "UUID_FMT"
> (%s)\n",
> +                            UUID_ARGS(&linport->header_.uuid),
> linport->name);
> +            }
> +
> +            const struct nbrec_logical_switch_port *loutport =
> +
> lsp_pair->outport;
> +            if (loutport) {
> +                ds_put_format(&ctx->output,
> +                          "            lsp-pair outport "UUID_FMT"
> (%s)\n",
> +                          UUID_ARGS(&loutport->header_.uuid),
> loutport->name);
> +            }
> +        }
> +    }
> +}
> +
> +static void
> +nbctl_lsp_chain_show(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_chain *lsp_chain;
> +
> +    if (ctx->argc == 2) {
> +        lsp_chain = lsp_chain_by_name_or_uuid(ctx, ctx->argv[1], false);
> +        if (lsp_chain) {
> +            print_lsp_chain(lsp_chain, ctx);
> +        }
> +    } else {
> +        NBREC_LOGICAL_PORT_CHAIN_FOR_EACH(lsp_chain, ctx->idl) {
> +            print_lsp_chain(lsp_chain, ctx);
> +        }
> +    }
> +}
> +/* End of port-chain operations */
> +
> +/*
> + * Port Pair Groups CLI Functions
> + */
> +static void
> +nbctl_lsp_pair_group_add(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_pair_group *lsp_pair_group;
> +    const char *ppg_name = ctx->argc >= 3 ? ctx->argv[2] : NULL;
> +
> +    const bool may_exist = shash_find(&ctx->options, "--may-exist") !=
> NULL;
> +    const bool add_duplicate = shash_find(&ctx->options,
> +                                           "--add-duplicate") != NULL;
> +    if (may_exist && add_duplicate) {
> +        ctl_fatal("--may-exist and --add-duplicate may not be used
> together");
> +    }
> +
> +    if (ppg_name) {
> +        if (!add_duplicate) {
> +            NBREC_LOGICAL_PORT_PAIR_GROUP_FOR_EACH(lsp_pair_group,
> ctx->idl) {
> +                if (!strcmp(lsp_pair_group->name, ppg_name)) {
> +                    if (may_exist) {
> +                        return;
> +                    }
> +                    ctl_fatal("%s: an lsp_port_pair_group with this \
> +                               name already exists", ppg_name);
> +                }
> +            }
> +        }
> +    } else if (may_exist) {
> +        ctl_fatal("--may-exist requires specifying a name");
> +    } else if (add_duplicate) {
> +        ctl_fatal("--add-duplicate requires specifying a name");
> +    }
> +
> +    /* check lsp_chain exists */
> +    const struct nbrec_logical_port_chain *lsp_chain;
> +    lsp_chain = lsp_chain_by_name_or_uuid(ctx, ctx->argv[1], true);
> +    if (!lsp_chain) {
> +        return;
> +    }
> +
> +    /* create the logical port-pair-group. */
> +    lsp_pair_group = nbrec_logical_port_pair_group_insert(ctx->txn);
> +    if (ppg_name) {
> +        nbrec_logical_port_pair_group_set_name(lsp_pair_group,
> ctx->argv[2]);
> +    }
> +
> +    int64_t sortkey = (int64_t) lsp_chain->n_port_pair_groups + 1;
> +    if (ctx->argc >= 4) {
> +         sortkey = (int64_t) atoi(ctx->argv[3]);
> +    }
> +    nbrec_logical_port_pair_group_set_sortkey(lsp_pair_group, &sortkey,
> 1);
> +
> +    /* Insert the logical port-pair-group into the logical switch. */
> +    nbrec_logical_port_chain_verify_port_pair_groups(lsp_chain);
> +    struct nbrec_logical_port_pair_group  **new_port_pair_group =
> +                              xmalloc(sizeof *new_port_pair_group *
> +                              (lsp_chain->n_port_pair_groups + 1));
> +    memcpy(new_port_pair_group, lsp_chain->port_pair_groups,
> +                  sizeof *new_port_pair_group *
> lsp_chain->n_port_pair_groups);
> +    new_port_pair_group[lsp_chain->n_port_pair_groups] =
> +        CONST_CAST(struct nbrec_logical_port_pair_group *,lsp_pair_group);
> +    nbrec_logical_port_chain_set_port_pair_groups(lsp_chain,
> +                     new_port_pair_group,
> +                     lsp_chain->n_port_pair_groups + 1);
> +    free(new_port_pair_group);
> +}
> +
> +/* Removes lsp-pair-group 'lsp_chain->port_pair_group[idx]'. */
> +static void
> +remove_lsp_pair_group(const struct nbrec_logical_port_chain *lsp_chain,
> +                      size_t idx)
> +{
> +    const struct nbrec_logical_port_pair_group *lsp_pair_group =
> +                                              lsp_chain->port_pair_groups[
> idx];
> +
> +    /* First remove 'lsp-pair-group' from the array of port-pair-groups.
> +     * This is what will actually cause the logical port-pair-group to be
> +     * deleted when the transaction is sent to the database server
> +     * (due to garbage collection). */
> +    struct nbrec_logical_port_pair_group **new_port_pair_group
> +        = xmemdup(lsp_chain->port_pair_groups,
> +            sizeof *new_port_pair_group * lsp_chain->n_port_pair_groups);
> +    new_port_pair_group[idx] =
> +            new_port_pair_group[lsp_chain->n_port_pair_groups - 1];
> +    nbrec_logical_port_chain_verify_port_pair_groups(lsp_chain);
> +    nbrec_logical_port_chain_set_port_pair_groups(lsp_chain,
> +            new_port_pair_group, lsp_chain->n_port_pair_groups - 1);
> +    free(new_port_pair_group);
> +
> +    /* Delete 'lsp-pair-group' from the IDL.  This won't have a real
> +     * effect on the database server (the IDL will suppress it in fact)
> but
> +     * it means that it won't show up when we iterate with
> +     * NBREC_LOGICAL_PORT_PAIR_GROUP_FOR_EACH later. */
> +    nbrec_logical_port_pair_group_delete(lsp_pair_group);
> +}
> +
> +static void
> +nbctl_lsp_pair_group_del(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_pair_group *lsp_pair_group;
> +    const bool must_exist = !shash_find(&ctx->options, "--if-exists");
> +
> +    lsp_pair_group = lsp_pair_group_by_name_or_uuid(ctx, ctx->argv[1],
> +                                                     must_exist);
> +    if (!lsp_pair_group) {
> +        return;
> +    }
> +
> +    /* Find the port-chain that contains 'port-pair-group', then delete
> it. */
> +    const struct nbrec_logical_port_chain *lsp_chain;
> +    NBREC_LOGICAL_PORT_CHAIN_FOR_EACH (lsp_chain, ctx->idl) {
> +        for (size_t i = 0; i < lsp_chain->n_port_pair_groups; i++) {
> +            if (lsp_chain->port_pair_groups[i] == lsp_pair_group) {
> +                remove_lsp_pair_group(lsp_chain,i);
> +                return;
> +            }
> +        }
> +    }
> +    if (must_exist) {
> +        ctl_fatal("logical port-pair-group %s is not part of any\
> +                    logical port-chain", ctx->argv[1]);
> +    }
> +}
> +
> +static void
> +nbctl_lsp_pair_group_list(struct ctl_context *ctx)
> +{
> +    const char *id = ctx->argv[1];
> +    const struct nbrec_logical_port_chain *lsp_chain;
> +    struct smap lsp_pair_groups;
> +    size_t i;
> +
> +    lsp_chain = lsp_chain_by_name_or_uuid(ctx, id, true);
> +    if (!lsp_chain) {
> +        return;
> +    }
> +
> +    smap_init(&lsp_pair_groups);
> +    for (i = 0; i < lsp_chain->n_port_pair_groups; i++) {
> +        const struct nbrec_logical_port_pair_group *lsp_pair_group =
> +                       lsp_chain->port_pair_groups[i];
> +        smap_add_format(&lsp_pair_groups, lsp_pair_group->name,
> +                         UUID_FMT " (%s)",
> +                         UUID_ARGS(&lsp_pair_group->header_.uuid),
> +                         lsp_pair_group->name);
> +    }
> +    const struct smap_node **nodes = smap_sort(&lsp_pair_groups);
> +    for (i = 0; i < smap_count(&lsp_pair_groups); i++) {
> +        const struct smap_node *node = nodes[i];
> +        ds_put_format(&ctx->output, "%s\n", node->value);
> +    }
> +    smap_destroy(&lsp_pair_groups);
> +    free(nodes);
> +}
> +
> +static void
> +nbctl_lsp_pair_group_add_port_pair(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_pair_group *lsp_pair_group;
> +    const struct nbrec_logical_port_pair *lsp_pair;
> +    const bool may_exist = shash_find(&ctx->options, "--may-exist") !=
> NULL;
> +
> +    lsp_pair_group = lsp_pair_group_by_name_or_uuid(ctx, ctx->argv[1],
> true);
> +    if (!lsp_pair_group) {
> +        return;
> +    }
> +
> +    /* Check that port-pair exists  */
> +    lsp_pair = lsp_pair_by_name_or_uuid(ctx, ctx->argv[2], true);
> +    if (!lsp_pair) {
> +        return;
> +    }
> +
> +    /* Do not add port pair more than once in a given port-pair-group */
> +    for (size_t i = 0; i < lsp_pair_group->n_port_pairs; i++) {
> +        if (lsp_pair_group->port_pairs[i] == lsp_pair) {
> +            if (!may_exist) {
> +                ctl_fatal("lsp_pair: %s is already added to\
> +                           port-pair-group %s\n", ctx->argv[2],
> ctx->argv[1]);
> +            }
> +            return;
> +        }
> +    }
> +
> +    /* Insert the logical port-pair into the logical port-pair-group. */
> +    nbrec_logical_port_pair_group_verify_port_pairs(lsp_pair_group);
> +    struct nbrec_logical_port_pair  **new_port_pair =
> +          xmalloc(sizeof *new_port_pair * (lsp_pair_group->n_port_pairs +
> 1));
> +    memcpy(new_port_pair, lsp_pair_group->port_pairs,
> +          sizeof *new_port_pair * lsp_pair_group->n_port_pairs);
> +    new_port_pair[lsp_pair_group->n_port_pairs] =
> +          CONST_CAST(struct nbrec_logical_port_pair *, lsp_pair);
> +    nbrec_logical_port_pair_group_set_port_pairs(lsp_pair_group,
> +          new_port_pair, lsp_pair_group->n_port_pairs + 1);
> +    free(new_port_pair);
> +}
> +
> +/* Removes port-pair from port-pair-groiup but does not delete it'. */
> +static void
> +remove_lsp_pair_from_port_pair_group(
> +      const struct nbrec_logical_port_pair_group *lsp_pair_group, size_t
> idx)
> +{
> +
> +    /* First remove 'lsp-pair' from the array of port-pairs.  This is
> +     * what will actually cause the logical port-pair to be deleted when
> the
> +     * transaction is sent to the database server
> +     * (due to garbage collection). */
> +    struct nbrec_logical_port_pair **new_port_pair
> +        = xmemdup(lsp_pair_group->port_pairs, sizeof *new_port_pair *
> +                  lsp_pair_group->n_port_pairs);
> +    new_port_pair[idx] = new_port_pair[lsp_pair_group->n_port_pairs - 1];
> +    nbrec_logical_port_pair_group_verify_port_pairs(lsp_pair_group);
> +    nbrec_logical_port_pair_group_set_port_pairs(lsp_pair_group,
> new_port_pair,
> +                  lsp_pair_group->n_port_pairs - 1);
> +    free(new_port_pair);
> +
> +    /* Do not delete actual port-pair as they are owned by a
> +     * lswitch and can be reused. */
> +}
> +
> +static void
> +nbctl_lsp_pair_group_del_port_pair(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_pair *lsp_pair;
> +    const struct nbrec_logical_port_pair_group *lsp_pair_group;
> +    const bool must_exist = !shash_find(&ctx->options, "--if-exists");
> +
> +
> +    lsp_pair_group = lsp_pair_group_by_name_or_uuid(ctx, ctx->argv[1],
> +                                                     must_exist);
> +    if (!lsp_pair_group) {
> +        return;
> +    }
> +    lsp_pair = lsp_pair_by_name_or_uuid(ctx, ctx->argv[2], must_exist);
> +    if (!lsp_pair) {
> +        return;
> +    }
> +    /* Find the port-pair_group that contains 'port-pair', then delete
> it. */
> +
> +    for (size_t i = 0; i < lsp_pair_group->n_port_pairs; i++) {
> +      if (lsp_pair_group->port_pairs[i] == lsp_pair) {
> +        remove_lsp_pair_from_port_pair_group(lsp_pair_group,i);
> +        return;
> +      }
> +    }
> +    if (must_exist) {
> +        ctl_fatal("logical port-pair %s is not part of any logical
> switch",
> +                  ctx->argv[1]);
> +    }
> +}
> +/* End of port-pair-group operations */
> +
> +/*
> + * Port Chain Classifier CLI Functions
> + */
> +static bool
> +parse_chain_direction(const char *direction)
> +{
> +    if (strcasecmp(direction, "uni-directional")) {
> +        return true;
> +    } else if (strcasecmp(direction, "bi-directional")) {
> +        return true;
> +    } else {
> +        ctl_fatal("%s: direction must be \"uni-directional\" \
> +                     or \"bi-directional\"", direction);
> +    }
> +}
> +static bool
> +parse_chain_path(const char *path){
> +    if (strcasecmp(path, "entry-lport")) {
> +        return true;
> +    } else if (strcasecmp(path, "exit-lport")) {
> +        return true;
> +    } else {
> +        ctl_fatal("%s: path must be \"entry-lport\" \
> +                     or \"exit-lport\"", path);
> +    }
> +}
> +static void
> +nbctl_lsp_chain_classifier_add(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_switch *lswitch;
> +    const struct nbrec_logical_port_chain *lsp_chain;
> +    const struct nbrec_logical_switch_port *lsp_input;
> +
> +    lswitch = ls_by_name_or_uuid(ctx, ctx->argv[1], true);
> +    lsp_chain = lsp_chain_by_name_or_uuid(ctx, ctx->argv[2], true);
> +    lsp_input = lsp_by_name_or_uuid(ctx, ctx->argv[3], true);
> +
> +    const char *lsp_chain_classifier_name =
> +                                       ctx->argc > 7 ? ctx->argv[7] :
> NULL;
> +
> +    const bool may_exist = shash_find(&ctx->options, "--may-exist") !=
> NULL;
> +    const bool add_duplicate = shash_find(&ctx->options,
> +                                 "--add-duplicate") != NULL;
> +    if (may_exist && add_duplicate) {
> +        ctl_fatal("--may-exist and --add-duplicate may not be used
> together");
> +    }
> +
> +    if (lsp_chain_classifier_name) {
> +        if (!add_duplicate) {
> +            const struct nbrec_logical_port_chain_classifier
> +
> *lsp_chain_classifier;
> +            NBREC_LOGICAL_PORT_CHAIN_CLASSIFIER_FOR_EACH(lsp_chain_
> classifier,
> +                                                  ctx->idl) {
> +                if (!strcmp(lsp_chain_classifier->name,
> +                                lsp_chain_classifier_name)) {
> +                    if (may_exist) {
> +                        return;
> +                    }
> +                    ctl_fatal("%s: an lsp_chain_classifier \
> +                               with this name already exists",
> +                               lsp_chain_classifier_name);
> +                }
> +            }
> +        }
> +    } else if (may_exist) {
> +        ctl_fatal("--may-exist requires specifying a name");
> +    } else if (add_duplicate) {
> +        ctl_fatal("--add-duplicate requires specifying a name");
> +    }
> +
> +    struct nbrec_logical_port_chain_classifier *lsp_chain_classifier;
> +    lsp_chain_classifier =
> +                       nbrec_logical_port_chain_
> classifier_insert(ctx->txn);
> +
> +    nbrec_logical_port_chain_classifier_set_chain(
> +                                        lsp_chain_classifier, lsp_chain);
> +    nbrec_logical_port_chain_classifier_set_port(
> +                                        lsp_chain_classifier, lsp_input);
> +    if (parse_chain_direction(ctx->argv[4])) {
> +    nbrec_logical_port_chain_classifier_set_direction(
> +                                        lsp_chain_classifier,
> ctx->argv[4]);
> +    }
> +    if (parse_chain_path(ctx->argv[5])) {
> +          nbrec_logical_port_chain_classifier_set_path(
> +                                        lsp_chain_classifier,
> ctx->argv[5]);
> +    }
> +    nbrec_logical_port_chain_classifier_set_match(
> +                                        lsp_chain_classifier,
> ctx->argv[6]);
> +    if (lsp_chain_classifier_name) {
> +           nbrec_logical_port_chain_classifier_set_name(lsp_chain_
> classifier,
> +                                        lsp_chain_classifier_name);
> +    }
> +    /* Insert the logical port-chain into the logical switch. */
> +    nbrec_logical_switch_verify_port_chain_classifiers(lswitch);
> +
> +    struct nbrec_logical_port_chain_classifier
> **new_port_chain_classifier =
> +               xmalloc( sizeof *new_port_chain_classifier *
> +              (lswitch->n_port_chain_classifiers + 1));
> +    memcpy(new_port_chain_classifier, lswitch->port_chain_classifiers,
> +              sizeof *new_port_chain_classifier *
> +              lswitch->n_port_chain_classifiers);
> +    new_port_chain_classifier[lswitch->n_port_chain_classifiers] =
> +              CONST_CAST(struct nbrec_logical_port_chain_classifier *,
> +                lsp_chain_classifier);
> +    nbrec_logical_switch_set_port_chain_classifiers(lswitch,
> +              new_port_chain_classifier,
> +              lswitch->n_port_chain_classifiers + 1);
> +    free(new_port_chain_classifier);
> +}
> +
> +/* Removes lsp-chain-classifier from logical switch. */
> +static void
> +remove_lsp_chain_classifier(const struct nbrec_logical_switch *lswitch,
> +                                size_t idx)
> +{
> +    const struct nbrec_logical_port_chain_classifier
> *lsp_chain_classifier =
> +                                      lswitch->port_chain_
> classifiers[idx];
> +
> +    /* First remove 'lsp-chain' from the array of port-chains.
> +     * This is what will actually cause the logical port-chain to be
> deleted
> +     * when the transaction is sent to the database server
> +     * (due to garbage collection). */
> +    struct nbrec_logical_port_chain_classifier
> **new_port_chain_classifier
> +        = xmemdup(lswitch->port_chain_classifiers,
> +                   sizeof *new_port_chain_classifier *
> +                    lswitch->n_port_chain_classifiers);
> +    new_port_chain_classifier[idx] =
> +             new_port_chain_classifier[lswitch->n_port_chain_classifiers
> - 1];
> +    nbrec_logical_switch_verify_port_chain_classifiers(lswitch);
> +    nbrec_logical_switch_set_port_chain_classifiers(lswitch,
> +                      new_port_chain_classifier,
> +                      lswitch->n_port_chain_classifiers - 1);
> +    free(new_port_chain_classifier);
> +
> +    /* Delete 'lsp-chain' from the IDL.  This won't have a real effect on
> the
> +     * database server (the IDL will suppress it in fact) but it means
> that it
> +     * won't show up when we iterate with
> +     * NBREC_LOGICAL_PORT_CHAIN_FOR_EACH later. */
> +    nbrec_logical_port_chain_classifier_delete(lsp_chain_classifier);
> +}
> +
> +static void
> +nbctl_lsp_chain_classifier_del(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_chain_classifier
> *lsp_chain_classifier;
> +    const bool must_exist = !shash_find(&ctx->options, "--if-exists");
> +
> +    lsp_chain_classifier =
> +         lsp_chain_classifier_by_name_or_uuid(ctx, ctx->argv[1],
> must_exist);
> +    if (!lsp_chain_classifier) {
> +        return;
> +    }
> +
> +    /* Find the lswitch that contains 'port-chain', then delete it. */
> +    const struct nbrec_logical_switch *lswitch;
> +    NBREC_LOGICAL_SWITCH_FOR_EACH (lswitch, ctx->idl) {
> +        for (size_t i = 0; i < lswitch->n_port_chain_classifiers; i++) {
> +            if (lswitch->port_chain_classifiers[i] ==
> lsp_chain_classifier) {
> +                remove_lsp_chain_classifier(lswitch,i);
> +                return;
> +            }
> +        }
> +    }
> +}
> +static void
> +print_lsp_chain_classifier_entry(struct ctl_context *ctx,
> +                      const struct nbrec_logical_switch *lswitch,
> +                      const char *chain_classifier_name_filter,
> +                      const bool show_switch_name)
> +{
> +    struct smap lsp_chain_classifiers;
> +    size_t i;
> +    /*
> +    * Loop over all chain classifiers
> +    */
> +    for (i = 0; i < lswitch->n_port_chain_classifiers; i++) {
> +        const struct nbrec_logical_port_chain_classifier
> +                 *lsp_chain_classifier =  lswitch->port_chain_
> classifiers[i];
> +        const struct nbrec_logical_port_chain *lsp_chain;
> +        const struct nbrec_logical_switch_port *lsp;
> +
> +        lsp_chain = lsp_chain_classifier->chain;
> +        lsp = lsp_chain_classifier->port;
> +        if (chain_classifier_name_filter &&
> +                 strcmp(chain_classifier_name_filter,
> +                 lsp_chain_classifier->name)) {
> +            continue;
> +        }
> +        if (show_switch_name) {
> +            smap_add_format(&lsp_chain_classifiers,
> +                            lsp_chain_classifier->name, UUID_FMT "
> (%s:%s)",
> +                            UUID_ARGS(&lsp_chain_
> classifier->header_.uuid),
> +                            lswitch->name, lsp_chain_classifier->name);
> +        } else {
> +            smap_add_format(&lsp_chain_classifiers,
> +                            lsp_chain_classifier->name, UUID_FMT " (%s)",
> +                            UUID_ARGS(&lsp_chain_
> classifier->header_.uuid),
> +                            lsp_chain_classifier->name);
> +        }
> +        smap_add_format(&lsp_chain_classifiers,
> +                            lsp_chain_classifier->name,UUID_FMT " (%s)",
> +                            UUID_ARGS(&lsp_chain->header_.uuid),
> +                            lsp_chain->name);
> +        smap_add_format(&lsp_chain_classifiers,
> +                            lsp_chain_classifier->name,UUID_FMT " (%s)",
> +                            UUID_ARGS(&lsp->header_.uuid),
> +                            lsp->name);
> +        smap_add_format(&lsp_chain_classifiers,
> +                            lsp_chain_classifier->name, "%s",
> +                            lsp_chain_classifier->direction);
> +        smap_add_format(&lsp_chain_classifiers,
> +                            lsp_chain_classifier->name, "%s",
> +                            lsp_chain_classifier->path);
> +        smap_add_format(&lsp_chain_classifiers,
> +                            lsp_chain_classifier->name, "%s",
> +                            lsp_chain_classifier->match);
> +    }
> +
> +    const struct smap_node **nodes = smap_sort(&lsp_chain_classifiers);
> +    for (i = 0; i < smap_count(&lsp_chain_classifiers); i++) {
> +        const struct smap_node *node = nodes[i];
> +        ds_put_format(&ctx->output, "%s\n", node->value);
> +    }
> +    smap_destroy(&lsp_chain_classifiers);
> +    free(nodes);
> +}
> +
> +static void
> +nbctl_lsp_chain_classifier_list(struct ctl_context *ctx)
> +{
> +    const char *id = ctx->argc > 1 ? ctx->argv[1] : NULL;
> +    const char *chain_classifier_name_filter =
> +                                        ctx->argc > 2 ? ctx->argv[2] :
> NULL;
> +    const struct nbrec_logical_switch *lswitch;
> +
> +    if (id) {
> +        lswitch = ls_by_name_or_uuid(ctx, id, true);
> +        print_lsp_chain_classifier_entry(ctx, lswitch,
> +                                 chain_classifier_name_filter, false);
> +    } else {
> +        NBREC_LOGICAL_SWITCH_FOR_EACH(lswitch, ctx->idl) {
> +            if (lswitch->n_port_chain_classifiers == 0) {
> +                continue;
> +            }
> +            print_lsp_chain_classifier_entry(ctx, lswitch,
> +                                  chain_classifier_name_filter, true);
> +        }
> +    }
> +}
> +/* End of port-chain-classifier operations  */
> +/*
> + * port-pair operations
> + */
> +static void
> +nbctl_lsp_pair_add(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_switch *lswitch;
> +    const struct nbrec_logical_switch_port *lsp_in,*lsp_out;
> +    const struct nbrec_logical_port_pair *lsp_pair;
> +
> +    const bool may_exist = shash_find(&ctx->options, "--may-exist") !=
> NULL;
> +    const bool add_duplicate = shash_find(&ctx->options,
> +                                           "--add-duplicate") != NULL;
> +
> +    lswitch = ls_by_name_or_uuid(ctx, ctx->argv[1], true);
> +    lsp_in = lsp_by_name_or_uuid(ctx, ctx->argv[2], true);
> +    lsp_out = lsp_by_name_or_uuid(ctx, ctx->argv[3], true);
> +
> +    const char *lsp_pair_name = ctx->argc >= 5 ? ctx->argv[4] : NULL;
> +    if (may_exist && add_duplicate) {
> +        ctl_fatal("--may-exist and --add-duplicate may not be used
> together");
> +    }
> +
> +    if (lsp_pair_name) {
> +        if (!add_duplicate) {
> +            NBREC_LOGICAL_PORT_PAIR_FOR_EACH(lsp_pair, ctx->idl) {
> +                if (!strcmp(lsp_pair->name, lsp_pair_name)) {
> +                    if (may_exist) {
> +                        return;
> +                    }
> +                    ctl_fatal("%s: an lsp_pair with this name already
> exists",
> +                              lsp_pair_name);
> +                }
> +            }
> +        }
> +    } else if (may_exist) {
> +        ctl_fatal("--may-exist requires specifying a name");
> +    } else if (add_duplicate) {
> +        ctl_fatal("--add-duplicate requires specifying a name");
> +    }
> +
> +    /* create the logical port-pair. */
> +    lsp_pair = nbrec_logical_port_pair_insert(ctx->txn);
> +    nbrec_logical_port_pair_set_inport(lsp_pair, lsp_in);
> +    nbrec_logical_port_pair_set_outport(lsp_pair, lsp_out);
> +    if (lsp_pair_name) {
> +        nbrec_logical_port_pair_set_name(lsp_pair, lsp_pair_name);
> +    }
> +
> +    /* Insert the logical port-pair into the logical port-pair-group. */
> +    nbrec_logical_switch_verify_port_pairs(lswitch);
> +    struct nbrec_logical_port_pair  **new_port_pair =
> +                 xmalloc(sizeof *new_port_pair * (lswitch->n_port_pairs +
> 1));
> +    memcpy(new_port_pair, lswitch->port_pairs,
> +                 sizeof *new_port_pair * lswitch->n_port_pairs);
> +    new_port_pair[lswitch->n_port_pairs] =
> +                 CONST_CAST(struct nbrec_logical_port_pair *, lsp_pair);
> +    nbrec_logical_switch_set_port_pairs(lswitch, new_port_pair,
> +                 lswitch->n_port_pairs + 1);
> +    free(new_port_pair);
> +}
> +/* Removes lswitch->pair_pair[idx]'. */
> +static void
> +remove_lsp_pair(const struct nbrec_logical_switch *lswitch, size_t idx)
> +{
> +    const struct nbrec_logical_port_pair *lsp_pair =
> lswitch->port_pairs[idx];
> +
> +    /* First remove 'lsp-pair' from the array of port-pairs.
> +     * This is what will actually cause the logical port-pair to be
> deleted
> +     * when the transaction is sent to the database server
> +     * (due to garbage collection). */
> +    struct nbrec_logical_port_pair **new_port_pair
> +        = xmemdup(lswitch->port_pairs,
> +                   sizeof *new_port_pair * lswitch->n_port_pairs);
> +    new_port_pair[idx] = new_port_pair[lswitch->n_port_pairs - 1];
> +    nbrec_logical_switch_verify_port_pairs(lswitch);
> +    nbrec_logical_switch_set_port_pairs(lswitch, new_port_pair,
> +                   lswitch->n_port_pairs - 1);
> +    free(new_port_pair);
> +
> +    /* Delete 'lsp-pair' from the IDL.  This won't have a real effect on
> the
> +     * database server (the IDL will suppress it in fact) but it means
> that it
> +     * won't show up when we iterate with
> +     * NBREC_LOGICAL_PORT_PAIR_FOR_EACH later. */
> +    nbrec_logical_port_pair_delete(lsp_pair);
> +}
> +
> +static void
> +nbctl_lsp_pair_del(struct ctl_context *ctx)
> +{
> +    const struct nbrec_logical_port_pair *lsp_pair;
> +    const bool must_exist = !shash_find(&ctx->options, "--if-exists");
> +
> +    lsp_pair = lsp_pair_by_name_or_uuid(ctx, ctx->argv[1], must_exist);
> +    if (!lsp_pair) {
> +        if (must_exist) {
> +            ctl_fatal("Cannot find lsp_pair: %s\n", ctx->argv[1]);
> +        }
> +    }
> +
> +    /* Find the port-pair_group that contains 'port-pair', then delete
> it. */
> +    const struct nbrec_logical_switch *lswitch;
> +    NBREC_LOGICAL_SWITCH_FOR_EACH(lswitch, ctx->idl) {
> +        for (size_t i = 0; i < lswitch->n_port_pairs; i++) {
> +            if (lswitch->port_pairs[i] == lsp_pair) {
> +                remove_lsp_pair(lswitch,i);
> +                return;
> +            }
> +        }
> +    }
> +    if (must_exist) {
> +        ctl_fatal("logical port-pair %s is not part of any logical
> switch",
> +                  ctx->argv[1]);
> +    }
> +}
> +
> +static void
> +print_lsp_pairs_for_switch(struct ctl_context *ctx,
> +                           const struct nbrec_logical_switch *lswitch,
> +                           const char *ppair_name_filter,
> +                           const bool show_switch_name)
> +{
> +    struct smap lsp_pairs;
> +    size_t i;
> +
> +    smap_init(&lsp_pairs);
> +    for (i = 0; i < lswitch->n_port_pairs; i++) {
> +        const struct nbrec_logical_port_pair *lsp_pair =
> +                    lswitch->port_pairs[i];
> +        if (ppair_name_filter!= NULL && strcmp(ppair_name_filter,
> +                   lsp_pair->name)!= 0) {
> +            continue;
> +        } else {
> +          const struct nbrec_logical_switch_port *linport  =
> lsp_pair->inport;
> +          const struct nbrec_logical_switch_port *loutport =
> lsp_pair->outport;
> +          const char *linport_name = linport ? linport->name :
> "<not_set>";
> +          const char *loutport_name = loutport ? loutport->name :
> "<not_set>";
> +
> +          if (show_switch_name) {
> +            smap_add_format(&lsp_pairs, lsp_pair->name,
> +                             UUID_FMT " (%s:%s) in:%s out:%s",
> +                             UUID_ARGS(&lsp_pair->header_.uuid),
> lswitch->name,
> +                             lsp_pair->name, linport_name, loutport_name);
> +          } else {
> +            smap_add_format(&lsp_pairs, lsp_pair->name,
> +                             UUID_FMT " (%s) in:%s out:%s",
> +                             UUID_ARGS(&lsp_pair->header_.uuid),
> +                             lsp_pair->name, linport_name, loutport_name);
> +        }
> +      }
> +    }
> +    const struct smap_node **nodes = smap_sort(&lsp_pairs);
> +    for (i = 0; i < smap_count(&lsp_pairs); i++) {
> +        const struct smap_node *node = nodes[i];
> +        ds_put_format(&ctx->output, "%s\n", node->value);
> +    }
> +    smap_destroy(&lsp_pairs);
> +    free(nodes);
> +}
> +
> +static void
> +nbctl_lsp_pair_list(struct ctl_context *ctx)
> +{
> +    const char *id = ctx->argc > 1 ? ctx->argv[1] : NULL;
> +    const char *pair_name_filter = ctx->argc > 2 ? ctx->argv[2] : NULL;
> +    const struct nbrec_logical_switch *lswitch;
> +    const struct nbrec_logical_port_pair *lsp_pair;
> +
> +    if (pair_name_filter!= NULL) {
> +      lsp_pair = lsp_pair_by_name_or_uuid(ctx, ctx->argv[2], true);
> +      if (!lsp_pair) {
> +           ctl_fatal("%s: an lsp_pair with this name does not exist",
> +                              ctx->argv[2]);
> +           return;
> +      }
> +    }
> +    if (id) {
> +        lswitch = ls_by_name_or_uuid(ctx, id, true);
> +        print_lsp_pairs_for_switch(ctx, lswitch, pair_name_filter,
> false);
> +    } else {
> +        NBREC_LOGICAL_SWITCH_FOR_EACH(lswitch, ctx->idl) {
> +            if (lswitch->n_port_pairs == 0) {
> +                continue;
> +            }
> +            print_lsp_pairs_for_switch(ctx, lswitch, pair_name_filter,
> true);
> +        }
> +    }
> +}
> +/* End of port-pair operations */
>
>  /* Returns the logical switch that contains 'lsp'. */
>  static const struct nbrec_logical_switch *
> @@ -1310,6 +2376,26 @@ nbctl_acl_add(struct ctl_context *ctx)
>          return;
>      }
>
> +    /* Validate ACL Options, if there were any provided. */
> +    struct smap acl_options = SMAP_INITIALIZER(&acl_options);
> +    if (ctx->argc >= 7) {
> +        struct sset acl_options_set;
> +        sset_from_delimited_string(&acl_options_set, ctx->argv[6], " ");
> +
> +        const char *acl_option_tuple;
> +        SSET_FOR_EACH (acl_option_tuple, &acl_options_set) {
> +            char *key, *value;
> +            value = xstrdup(acl_option_tuple);
> +            key = strsep(&value, "=");
> +            if (value) {
> +                smap_add(&acl_options, key, value);
> +            }
> +            free(key);
> +        }
> +
> +        sset_destroy(&acl_options_set);
> +    }
> +
>      /* Create the acl. */
>      struct nbrec_acl *acl = nbrec_acl_insert(ctx->txn);
>      nbrec_acl_set_priority(acl, priority);
> @@ -1339,6 +2425,8 @@ nbctl_acl_add(struct ctl_context *ctx)
>      new_acls[ls->n_acls] = acl;
>      nbrec_logical_switch_set_acls(ls, new_acls, ls->n_acls + 1);
>      free(new_acls);
> +
> +    smap_destroy(&acl_options);
>  }
>
>  static void
> @@ -3293,6 +4381,8 @@ static const struct ctl_command_syntax
> nbctl_commands[] = {
>      { "sync", 0, 0, "", nbctl_pre_sync, nbctl_sync, NULL, "", RO },
>      { "show", 0, 1, "[SWITCH]", NULL, nbctl_show, NULL, "", RO },
>
> +
> +
>      /* logical switch commands. */
>      { "ls-add", 0, 1, "[SWITCH]", NULL, nbctl_ls_add, NULL,
>        "--may-exist,--add-duplicate", RW },
> @@ -3339,6 +4429,49 @@ static const struct ctl_command_syntax
> nbctl_commands[] = {
>      { "lsp-get-dhcpv4-options", 1, 1, "PORT", NULL,
>        nbctl_lsp_get_dhcpv4_options, NULL, "", RO },
>
> +          /* lsp-chain-classifier commands. */
> +    { "lsp-chain-classifier-add", 6, 7,
> +                       "SWITCH, CHAIN, PORT, DIRECTION, PATH, MATCH,
> [NAME]",
> +      NULL, nbctl_lsp_chain_classifier_add, NULL,
> +                       "--may-exist,--add-duplicate", RW },
> +    { "lsp-chain-classifier-del", 1, 1, "CHAIN", NULL,
> +      nbctl_lsp_chain_classifier_del, NULL, "--if-exists", RW },
> +    { "lsp-chain-classifier-list", 1, 2, "SWITCH, [CHAIN]", NULL,
> +      nbctl_lsp_chain_classifier_list, NULL, "", RO },
> +
> +    /* lsp-chain commands. */
> +    { "lsp-chain-add", 1, 2, "SWITCH [CHAIN]", NULL,
> +      nbctl_lsp_chain_add,
> +      NULL, "--may-exist,--add-duplicate", RW },
> +    { "lsp-chain-del", 1, 1, "CHAIN", NULL, nbctl_lsp_chain_del,
> +      NULL, "--if-exists", RW },
> +    { "lsp-chain-list", 0, 2, "[SWITCH [CHAIN]]", NULL,
> +      nbctl_lsp_chain_list, NULL, "", RO },
> +    { "lsp-chain-show", 0, 1, "[CHAIN]", NULL,
> +      nbctl_lsp_chain_show, NULL, "", RO },
> +
> +    /* lsp-pair-group commands. */
> +    { "lsp-pair-group-add", 1, 3, "CHAIN [PAIR-GROUP [OFFSET]]",
> +      NULL, nbctl_lsp_pair_group_add, NULL, "--may-exist,--add-duplicate",
> +      RW },
> +    { "lsp-pair-group-del", 1, 1, "PAIR-GROUP", NULL,
> nbctl_lsp_pair_group_del,
> +      NULL, "--if-exists", RW },
> +    { "lsp-pair-group-list", 1, 1, "CHAIN", NULL,
> nbctl_lsp_pair_group_list,
> +      NULL, "", RO },
> +    { "lsp-pair-group-add-port-pair", 2, 2, "PAIR-GROUP LSP-PAIR",
> +      NULL, nbctl_lsp_pair_group_add_port_pair, NULL, "--may-exist", RW
> },
> +    { "lsp-pair-group-del-port-pair", 2, 2, "PAIR-GROUP LSP-PAIR",
> +      NULL, nbctl_lsp_pair_group_del_port_pair, NULL, "--if-exists", RW
> },
> +
> +    /* lsp-pair commands. */
> +    { "lsp-pair-add", 3, 4, "SWITCH, PORT-IN, PORT-OUT [LSP-PAIR]", NULL,
> +      nbctl_lsp_pair_add,
> +      NULL, "--may-exist,--add-duplicate", RW },
> +    { "lsp-pair-del", 1, 1, "LSP-PAIR", NULL, nbctl_lsp_pair_del,
> +      NULL, "--if-exists", RW },
> +    { "lsp-pair-list", 0, 2, "[SWITCH [LSP-PAIR]]", NULL,
> +      nbctl_lsp_pair_list, NULL, "", RO },
> +
>      /* logical router commands. */
>      { "lr-add", 0, 1, "[ROUTER]", NULL, nbctl_lr_add, NULL,
>        "--may-exist,--add-duplicate", RW },
>
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>


More information about the dev mailing list