[ovs-dev] [PATCH ovn v4] OVN: Multiple distributed gateway port support

Dhathri Purohith dhathri.purohith at nutanix.com
Wed Mar 24 20:27:07 UTC 2021


Hi Numan, 

Thanks a lot for the review.
Will work on the ddlog changes and update the patch.

Thanks,
Dhathri

On 3/24/21, 11:43 AM, "Numan Siddique" <numans at ovn.org> wrote:

    On Thu, Mar 4, 2021 at 6:49 AM svc.eng.git-patch at nutanix.com
    <svc.eng.git-patch at nutanix.com> wrote:
    >
    > From: Ankur Sharma <ankurmnnit2004 at gmail.com>
    >
    > By default, OVN support only one distributed gateway
    > port (we will call it l3dgw port for further reference)
    > per logical router. While a single l3dgw port suffices
    > for most of the North South connectivity, however there
    > are requirements where a logical router could be connected
    > to multiple physical networks and based on routing decision
    > packet could go to vlan X or vlan Y. Additionally, packet
    > may or may not get NATed based on the configuration.
    >
    > This patch adds flexibility of having multiple l3dgw ports
    > per logical router.
    >
    > Changes can classified as following:
    > a. Data structure changes to allow multiple l3dgw ports per
    >    ovn_datapath.
    >
    > b. Consumption of new data structure in logical flows for
    >    individual features.
    >
    > c. Features that require changes are:
    >    i. Regular NS traffic flow.
    >   ii. Network Address Translation.
    >  iii. Load Balancer
    >   iv. Gateway_mtu.
    >    v. reside-on-redirect-chassis
    >   vi. Misc code sections that assumed a single l3dgw port.
    >
    > d. ovn-nbctl cli change to allow multiple external ips
    >    for a logical ip for same type.
    >
    > e. Except for reside-on-redirect-chassis all the other features
    >    could be extended to multiple l3dgw ports. Reside on redirect
    >    chassis with its current specification could not be extended
    >    and hence should be used only with the logical router that
    >    has a single l3dgw port.
    >
    > FUTURE WORK:
    > CT ZONES are still common for traffic from different physical networks.
    > This adds a restriction/assumption that same 5 tuple will not come
    > from different l3dgw ports.
    > A cleaner approach would be have different ct zones for each l3dgw port.
    > Changing the CT ZONE assignment is one of the enhancements we are
    > considering as next step.
    >
    > Signed-off-by: Ankur Sharma <ankurmnnit2004 at gmail.com>
    > Signed-off-by: Dhathri Purohith <dhathri.purohith at nutanix.com>
    > Co-authored-by: Dhathri Purohith <dhathri.purohith at nutanix.com>

    Hi Ankur,

    Thanks for v4.  I'm sorry that it took a while to review this patch.

    This patch needs a rebase. Overall the patch LGTM.  I have few comments.
    Please see below. If you can address those and add ddlog support in
    v5, I think it is good to go.
    )
    This patch needs to update the documentation for distributed gateway
    ports in ovn-nb.xml
    (mainly here - https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ovn-2Dorg_ovn_blob_master_ovn-2Dnb.xml-23L2274&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=dftLCwr11LPmmIAnjdakaot5N4f-Q2Qfwcs7XdgQZXE&m=qsTaQmkBjQHuap5egxV8-PeoiNS8HU-jA0qbn-WCVpQ&s=_6BAZU3G-YnVLUvokLWftq8eU7xeE9IAMIPBsOYF_eQ&e= 
    and in ovn-architecture.7.xml.  Please do consider adding a section
    for multiple gw ports
    in ovn-architecture.7.xml if you can.

    Please add an entry in the NEWS file about this feature.

    Please see below for additional comments.

    > ---
    >  northd/ovn-northd.c   | 521 +++++++++++++++++++++++++++---------------
    >  ovn-nb.xml            |   4 +-
    >  tests/ovn-nbctl.at    |  38 ++-
    >  tests/ovn-northd.at   | 465 ++++++++++++++++++++++++++++++++++++-
    >  tests/ovn.at          | 310 ++++++++++++++++++++++++-
    >  utilities/ovn-nbctl.c | 147 +++++++++++-
    >  6 files changed, 1276 insertions(+), 209 deletions(-)
    >
    > diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c
    > index ac872aade..cc228fd96 100644
    > --- a/northd/ovn-northd.c
    > +++ b/northd/ovn-northd.c
    > @@ -602,6 +602,19 @@ ovn_mcast_group_allocate_key(struct mcast_info *mcast_info)
    >                                &mcast_info->group_tnlid_hint);
    >  }
    >
    > +struct ovn_l3dgw_port {
    > +    /* OVN northd only needs to know about the logical router gateway port for
    > +     * NAT on a distributed router.  This "distributed gateway port" is
    > +     * populated only when there is a gateway chassis or ha chassis group
    > +     * specified for one of the ports on the logical router. Otherwise this
    > +     * will be NULL. */
    > +    struct ovn_port *dgw_port;
    > +
    > +    /* The "derived" OVN port representing the instance of l3dgw_port on
    > +     * the gateway chassis. */
    > +    struct ovn_port *redirect_port;
    > +};
    > +
    >  /* The 'key' comes from nbs->header_.uuid or nbr->header_.uuid or
    >   * sb->external_ids:logical-switch. */
    >  struct ovn_datapath {
    > @@ -633,14 +646,9 @@ struct ovn_datapath {
    >      /* Multicast data. */
    >      struct mcast_info mcast_info;
    >
    > -    /* OVN northd only needs to know about the logical router gateway port for
    > -     * NAT on a distributed router.  This "distributed gateway port" is
    > -     * populated only when there is a gateway chassis specified for one of
    > -     * the ports on the logical router.  Otherwise this will be NULL. */
    > -    struct ovn_port *l3dgw_port;
    > -    /* The "derived" OVN port representing the instance of l3dgw_port on
    > -     * the gateway chassis. */
    > -    struct ovn_port *l3redirect_port;
    > +    /* L3 distributed gateway ports */
    > +    struct ovn_l3dgw_port *l3dgw_ports;
    > +    size_t n_l3dgw_ports;
    >
    >      /* NAT entries configured on the router. */
    >      struct ovn_nat *nat_entries;
    > @@ -863,6 +871,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct ovn_datapath *od)
    >          ovn_destroy_tnlids(&od->port_tnlids);
    >          destroy_ipam_info(&od->ipam_info);
    >          free(od->router_ports);
    > +        free(od->l3dgw_ports);
    >          destroy_nat_entries(od);
    >          free(od->nat_entries);
    >          free(od->localnet_ports);
    > @@ -1416,6 +1425,82 @@ struct ovn_port {
    >      struct ovs_list list;       /* In list of similar records. */
    >  };
    >
    > +/* Get the l3dgw port corresponding to a logical router port.*/
    > +static inline struct ovn_l3dgw_port*

    Is there a need for inline here ? I'd suggest removing since the
    function is defined
    in the source file.

    > +ovn_get_l3dgw_port_from_lrp(const struct ovn_port *op)
    > +{
    > +    struct ovn_datapath *od = op->od;
    > +
    > +    if (!op || !op->nbrp) {

    This seems odd. Maybe you want to move this if condition up before
    accessing op->od.  If op is NULL, then the first
    statement op->od will crash.

    > +        return NULL;
    > +    }
    > +
    > +    for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
    > +        struct ovn_l3dgw_port *l3dgw_port =
    > +            &(od->l3dgw_ports[i]);
    > +        if (op == l3dgw_port->dgw_port) {
    > +            return l3dgw_port;
    > +        }
    > +    }
    > +
    > +    return NULL;
    > +}
    > +
    > +/* Get the l3dgw port corresponding to a logical router port
    > + * with input ip */
    > +static struct ovn_l3dgw_port*
    > +ovn_get_l3dgw_port_from_ip(struct ovn_datapath *od, struct in6_addr ip_addr)
    > +{
    > +    if (!od || !od->nbr) {
    > +        return NULL;
    > +    }
    > +
    > +    for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
    > +        struct ovn_l3dgw_port *l3dgw_port =
    > +                                       &(od->l3dgw_ports[i]);
    > +        struct ovn_port *op = l3dgw_port->dgw_port;
    > +        struct lport_addresses lrp_networks;
    > +
    > +        if (!extract_lrp_networks(op->nbrp, &lrp_networks)) {

    I didn't understand why you're trying to extract the lrp networks given that
    op->lrp_networks would have the same information.  I think you can just
    run the for loop on op->lrp_networks.


    > +            static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
    > +            VLOG_WARN_RL(&rl, "Extract addresses failed.");
    > +            continue;
    > +        }
    > +
    > +        bool is_v4 = IN6_IS_ADDR_V4MAPPED(&ip_addr);
    > +        if (!is_v4) {
    > +            for (int j = 0; j < lrp_networks.n_ipv6_addrs; j++) {
    > +                struct ipv6_netaddr *lrp6_addr =
    > +                                    &(lrp_networks.ipv6_addrs[j]);
    > +                struct in6_addr ip6_mask = ipv6_addr_bitand(&lrp6_addr->mask,
    > +                                                            &ip_addr);
    > +
    > +                if (ipv6_addr_equals(&ip6_mask, &(lrp6_addr->network))) {
    > +                    destroy_lport_addresses(&lrp_networks);
    > +                    return l3dgw_port;
    > +                }
    > +            }
    > +        } else {
    > +            for (int j = 0; j < lrp_networks.n_ipv4_addrs; j++) {
    > +                struct ipv4_netaddr *lrp4_addr =
    > +                                    &(lrp_networks.ipv4_addrs[j]);
    > +                uint32_t addr = ntohl(lrp4_addr->addr);
    > +                uint32_t network = ntohl(lrp4_addr->network);
    > +                uint32_t mask4 = ntohl(lrp4_addr->mask);
    > +                uint32_t bcast = addr | ~mask4;
    > +                uint32_t ip4 = ntohl(in6_addr_get_mapped_ipv4(&ip_addr));
    > +
    > +                if (ip4 >= network && ip4 < bcast) {
    > +                    destroy_lport_addresses(&lrp_networks);
    > +                    return l3dgw_port;
    > +                }
    > +            }
    > +        }
    > +    }
    > +
    > +    return NULL;
    > +}
    > +
    >  static void
    >  ovn_port_set_nb(struct ovn_port *op,
    >                  const struct nbrec_logical_switch_port *nbsp,
    > @@ -2327,14 +2412,6 @@ join_logical_ports(struct northd_context *ctx,
    >                                       "on L3 gateway router", nbrp->name);
    >                          continue;
    >                      }
    > -                    if (od->l3dgw_port || od->l3redirect_port) {
    > -                        static struct vlog_rate_limit rl
    > -                            = VLOG_RATE_LIMIT_INIT(1, 1);
    > -                        VLOG_WARN_RL(&rl, "Bad configuration: multiple "
    > -                                     "distributed gateway ports on logical "
    > -                                     "router %s", od->nbr->name);
    > -                        continue;
    > -                    }
    >
    >                      char *redirect_name =
    >                          ovn_chassis_redirect_name(nbrp->name);
    > @@ -2355,8 +2432,12 @@ join_logical_ports(struct northd_context *ctx,
    >
    >                      /* Set l3dgw_port and l3redirect_port in od, for later
    >                       * use during flow creation. */
    > -                    od->l3dgw_port = op;
    > -                    od->l3redirect_port = crp;
    > +                    od->l3dgw_ports = xrealloc(od->l3dgw_ports,
    > +                                               sizeof *od->l3dgw_ports *
    > +                                               (od->n_l3dgw_ports + 1));
    > +                    od->l3dgw_ports[od->n_l3dgw_ports].dgw_port = op;
    > +                    od->l3dgw_ports[od->n_l3dgw_ports].redirect_port = crp;
    > +                    od->n_l3dgw_ports++;
    >                  }
    >              }
    >          }
    > @@ -2512,7 +2593,7 @@ get_nat_addresses(const struct ovn_port *op, size_t *n)
    >
    >          /* Determine whether this NAT rule satisfies the conditions for
    >           * distributed NAT processing. */
    > -        if (op->od->l3redirect_port && !strcmp(nat->type, "dnat_and_snat")
    > +        if (op->od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat")
    >              && nat->logical_port && nat->external_mac) {
    >              /* Distributed NAT rule. */
    >              if (eth_addr_from_string(nat->external_mac, &mac)) {
    > @@ -2572,11 +2653,13 @@ get_nat_addresses(const struct ovn_port *op, size_t *n)
    >      sset_destroy(&all_ips_v6);
    >
    >      if (central_ip_address) {
    > +        struct ovn_l3dgw_port *l3dgw_port =
    > +            ovn_get_l3dgw_port_from_lrp(op);
    >          /* Gratuitous ARP for centralized NAT rules on distributed gateway
    >           * ports should be restricted to the gateway chassis. */
    > -        if (op->od->l3redirect_port) {
    > +        if (l3dgw_port) {
    >              ds_put_format(&c_addresses, " is_chassis_resident(%s)",
    > -                          op->od->l3redirect_port->json_key);
    > +                          l3dgw_port->redirect_port->json_key);
    >          }
    >
    >          addresses[n_nats++] = ds_steal_cstr(&c_addresses);
    > @@ -3075,7 +3158,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
    >              char **nats = NULL;
    >              if (nat_addresses && !strcmp(nat_addresses, "router")) {
    >                  if (op->peer && op->peer->od
    > -                    && (chassis || op->peer->od->l3redirect_port)) {
    > +                    && (chassis || op->peer->od->n_l3dgw_ports)) {
    >                      nats = get_nat_addresses(op->peer, &n_nats);
    >                  }
    >              /* Only accept manual specification of ethernet address
    > @@ -3111,12 +3194,13 @@ ovn_port_update_sbrec(struct northd_context *ctx,
    >               * sending the GARPs for the router port IPs.
    >               * */
    >              bool add_router_port_garp = false;
    > -            if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_port &&
    > -                op->peer->od->l3redirect_port &&
    > -                (smap_get_bool(&op->peer->nbrp->options,
    > -                              "reside-on-redirect-chassis", false) ||
    > -                op->peer == op->peer->od->l3dgw_port)) {
    > +            struct ovn_l3dgw_port *l3dgw_port = NULL;
    > +            if (op->peer && op->peer->nbrp && op->peer->od->n_l3dgw_ports) {
    > +              l3dgw_port = ovn_get_l3dgw_port_from_lrp(op->peer);
    > +              if (smap_get_bool(&op->peer->nbrp->options,
    > +                  "reside-on-redirect-chassis", false) || l3dgw_port) {
    >                  add_router_port_garp = true;
    > +              }
    >              } else if (chassis && op->od->n_localnet_ports) {
    >                  add_router_port_garp = true;
    >              }
    > @@ -3130,9 +3214,12 @@ ovn_port_update_sbrec(struct northd_context *ctx,
    >                                    op->peer->lrp_networks.ipv4_addrs[i].addr_s);
    >                  }
    >
    > -                if (op->peer->od->l3redirect_port) {
    > +                if (op->peer->od->n_l3dgw_ports == 1) {
    > +                    if (!l3dgw_port) {
    > +                        l3dgw_port = &op->peer->od->l3dgw_ports[0];
    > +                    }
    >                      ds_put_format(&garp_info, " is_chassis_resident(%s)",
    > -                                  op->peer->od->l3redirect_port->json_key);
    > +                                  l3dgw_port->redirect_port->json_key);
    >                  }
    >
    >                  n_nats++;
    > @@ -6156,13 +6243,16 @@ build_lrouter_groups__(struct hmap *ports, struct ovn_datapath *od)
    >  {
    >      ovs_assert((od && od->nbr && od->lr_group));
    >
    > -    if (od->l3dgw_port && od->l3redirect_port) {
    > +    for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
    > +        struct ovn_l3dgw_port *l3dgw_port =
    > +            &(od->l3dgw_ports[i]);
    > +
    >          /* It's a logical router with gateway port. If it
    >           * has HA_Chassis_Group associated to it in SB DB, then store the
    >           * ha chassis group name. */
    > -        if (od->l3redirect_port->sb->ha_chassis_group) {
    > +        if (l3dgw_port->redirect_port->sb->ha_chassis_group) {
    >              sset_add(&od->lr_group->ha_chassis_groups,
    > -                     od->l3redirect_port->sb->ha_chassis_group->name);
    > +                     l3dgw_port->redirect_port->sb->ha_chassis_group->name);
    >          }
    >      }
    >
    > @@ -7413,24 +7503,28 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
    >                  ds_clear(match);
    >                  ds_put_format(match, "eth.dst == "ETH_ADDR_FMT,
    >                                ETH_ADDR_ARGS(mac));
    > -                if (op->peer->od->l3dgw_port
    > -                    && op->peer->od->l3redirect_port
    > -                    && op->od->n_localnet_ports) {
    > +                if (op->peer->od->n_l3dgw_ports &&
    > +                    op->od->n_localnet_ports) {
    >                      bool add_chassis_resident_check = false;
    > -                    if (op->peer == op->peer->od->l3dgw_port) {
    > +                    struct ovn_l3dgw_port *l3dgw_port =
    > +                        ovn_get_l3dgw_port_from_lrp(op->peer);
    > +                    if (l3dgw_port) {
    >                          /* The peer of this port represents a distributed
    >                           * gateway port. The destination lookup flow for the
    >                           * router's distributed gateway port MAC address should
    >                           * only be programmed on the gateway chassis. */
    >                          add_chassis_resident_check = true;
    > -                    } else {
    > +                    } else if (op->peer->od->n_l3dgw_ports == 1) {
    >                          /* Check if the option 'reside-on-redirect-chassis'
    >                           * is set to true on the peer port. If set to true
    >                           * and if the logical switch has a localnet port, it
    >                           * means the router pipeline for the packets from
    >                           * this logical switch should be run on the chassis
    >                           * hosting the gateway port.
    > +                         * 'reside-on-redirect-chassis' is supported only for
    > +                         * logical routers with single l3dgw port.
    >                           */
    > +                        l3dgw_port = &op->peer->od->l3dgw_ports[0];
    >                          add_chassis_resident_check = smap_get_bool(
    >                              &op->peer->nbrp->options,
    >                              "reside-on-redirect-chassis", false);
    > @@ -7438,7 +7532,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
    >
    >                      if (add_chassis_resident_check) {
    >                          ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                      op->peer->od->l3redirect_port->json_key);
    > +                                      l3dgw_port->redirect_port->json_key);
    >                      }
    >                  }
    >
    > @@ -7451,8 +7545,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
    >
    >                  /* Add ethernet addresses specified in NAT rules on
    >                   * distributed logical routers. */
    > -                if (op->peer->od->l3dgw_port
    > -                    && op->peer == op->peer->od->l3dgw_port) {
    > +                if (ovn_get_l3dgw_port_from_lrp(op->peer)) {
    >                      for (int j = 0; j < op->peer->od->nbr->n_nat; j++) {
    >                          const struct nbrec_nat *nat
    >                                                    = op->peer->od->nbr->nat[j];
    > @@ -8583,33 +8676,49 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od,
    >      build_empty_lb_event_flow(od, lflows, lb_vip, lb, S_ROUTER_IN_DNAT,
    >                                meter_groups);
    >
    > -    /* A match and actions for new connections. */
    > -    char *new_match = xasprintf("ct.new && %s", ds_cstr(match));
    > -    if (lb_force_snat_ip) {
    > -        char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s",
    > -                                      ds_cstr(actions));
    > -        ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > -                                new_match, new_actions, &lb->header_);
    > -        free(new_actions);
    > -    } else {
    > -        ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > -                                new_match, ds_cstr(actions), &lb->header_);
    > -    }
    > +    for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
    > +        struct ovn_l3dgw_port *l3dgw_port = &(od->l3dgw_ports[i]);
    >
    > -    /* A match and actions for established connections. */
    > -    char *est_match = xasprintf("ct.est && %s", ds_cstr(match));
    > -    if (lb_force_snat_ip) {
    > -        ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > -                                est_match,
    > -                                "flags.force_snat_for_lb = 1; ct_dnat;",
    > -                                &lb->header_);
    > -    } else {
    > -        ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > -                                est_match, "ct_dnat;", &lb->header_);
    > -    }
    > +        char *new_match = NULL;
    > +        if (lb_vip->n_backends || !lb_vip->empty_backend_rej) {
    > +          /* A match and actions for new connections. */
    > +          new_match = xasprintf("ct.new && %s && inport == %s && "
    > +                                "is_chassis_resident(%s)", ds_cstr(match),
    > +                                l3dgw_port->dgw_port->json_key,
    > +                                l3dgw_port->redirect_port->json_key);
    > +        } else {
    > +          new_match = xasprintf("ct.new && %s", ds_cstr(match));
    > +        }
    >
    > -    free(new_match);
    > -    free(est_match);
    > +        if (lb_force_snat_ip) {
    > +            char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s",
    > +                                          ds_cstr(actions));
    > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > +                                    new_match, new_actions, &lb->header_);
    > +            free(new_actions);
    > +        } else {
    > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > +                                    new_match, ds_cstr(actions), &lb->header_);
    > +        }
    > +
    > +        /* A match and actions for established connections. */
    > +        char *est_match = xasprintf("ct.est && %s && inport == %s && "
    > +                                    "is_chassis_resident(%s)", ds_cstr(match),
    > +                                    l3dgw_port->dgw_port->json_key,
    > +                                    l3dgw_port->redirect_port->json_key);
    > +        if (lb_force_snat_ip) {
    > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > +                                    est_match,
    > +                                    "flags.force_snat_for_lb = 1; ct_dnat;",
    > +                                    &lb->header_);
    > +        } else {
    > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
    > +                                    est_match, "ct_dnat;", &lb->header_);
    > +        }
    > +
    > +        free(new_match);
    > +        free(est_match);
    > +    }
    >
    >      const char *ip_match = NULL;
    >      if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
    > @@ -8644,49 +8753,55 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od,
    >          ds_destroy(&unsnat_match);
    >      }
    >
    > -    if (!od->l3dgw_port || !od->l3redirect_port || !lb_vip->n_backends) {
    > +    if (!od->n_l3dgw_ports || !lb_vip->n_backends) {
    >          return;
    >      }
    >
    > -    /* Add logical flows to UNDNAT the load balanced reverse traffic in
    > -     * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the logical
    > -     * router has a gateway router port associated.
    > -     */
    > -    struct ds undnat_match = DS_EMPTY_INITIALIZER;
    > -    ds_put_format(&undnat_match, "%s && (", ip_match);
    > +    for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
    >
    > -    for (size_t i = 0; i < lb_vip->n_backends; i++) {
    > -        struct ovn_lb_backend *backend = &lb_vip->backends[i];
    > -        ds_put_format(&undnat_match, "(%s.src == %s", ip_match,
    > -                      backend->ip_str);
    > +        /* Add logical flows to UNDNAT the load balanced reverse traffic in
    > +         * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the
    > +         * logical router has a gateway router port associated.
    > +         */
    > +        struct ds undnat_match = DS_EMPTY_INITIALIZER;
    > +        ds_put_format(&undnat_match, "%s && (", ip_match);
    > +
    > +        struct ovn_l3dgw_port *l3dgw_port = &od->l3dgw_ports[i];
    > +
    > +        for (size_t j = 0; j < lb_vip->n_backends; j++) {
    > +            struct ovn_lb_backend *backend = &lb_vip->backends[j];
    > +            ds_put_format(&undnat_match, "(%s.src == %s", ip_match,
    > +                          backend->ip_str);
    >
    > -        if (backend->port) {
    > -            ds_put_format(&undnat_match, " && %s.src == %d) || ",
    > -                          proto, backend->port);
    > +            if (backend->port) {
    > +                ds_put_format(&undnat_match, " && %s.src == %d) || ",
    > +                              proto, backend->port);
    > +            } else {
    > +                ds_put_cstr(&undnat_match, ") || ");
    > +            }
    > +        }
    > +
    > +        ds_chomp(&undnat_match, ' ');
    > +        ds_chomp(&undnat_match, '|');
    > +        ds_chomp(&undnat_match, '|');
    > +        ds_chomp(&undnat_match, ' ');
    > +        ds_put_format(&undnat_match, ") && outport == %s && "
    > +                      "is_chassis_resident(%s)",
    > +                      l3dgw_port->dgw_port->json_key,
    > +                      l3dgw_port->redirect_port->json_key);
    > +        if (lb_force_snat_ip) {
    > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
    > +                    ds_cstr(&undnat_match),
    > +                    "flags.force_snat_for_lb = 1; ct_dnat;",
    > +                    &lb->header_);
    >          } else {
    > -            ds_put_cstr(&undnat_match, ") || ");
    > -        }
    > -    }
    > -
    > -    ds_chomp(&undnat_match, ' ');
    > -    ds_chomp(&undnat_match, '|');
    > -    ds_chomp(&undnat_match, '|');
    > -    ds_chomp(&undnat_match, ' ');
    > -    ds_put_format(&undnat_match, ") && outport == %s && "
    > -                 "is_chassis_resident(%s)", od->l3dgw_port->json_key,
    > -                 od->l3redirect_port->json_key);
    > -    if (lb_force_snat_ip) {
    > -        ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
    > -                                ds_cstr(&undnat_match),
    > -                                "flags.force_snat_for_lb = 1; ct_dnat;",
    > -                                &lb->header_);
    > -    } else {
    > -        ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
    > -                                ds_cstr(&undnat_match), "ct_dnat;",
    > -                                &lb->header_);
    > -    }
    > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
    > +                    ds_cstr(&undnat_match), "ct_dnat;",
    > +                    &lb->header_);
    > +        }
    >
    > -    ds_destroy(&undnat_match);
    > +        ds_destroy(&undnat_match);
    > +    }
    >  }
    >
    >  #define ND_RA_MAX_INTERVAL_MAX 1800
    > @@ -8808,7 +8923,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
    >  {
    >      struct nbrec_address_set *allowed_ext_ips = nat->allowed_ext_ips;
    >      struct nbrec_address_set *exempted_ext_ips = nat->exempted_ext_ips;
    > -    bool is_gw_router = !od->l3dgw_port;
    > +    bool is_gw_router = !od->n_l3dgw_ports;
    >
    >      ovs_assert(allowed_ext_ips || exempted_ext_ips);
    >
    > @@ -9023,9 +9138,11 @@ build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op,
    >           * upstream MAC learning points to the gateway chassis.
    >           * Also need to avoid generation of multiple ARP responses
    >           * from different chassis. */
    > -        if (op->od->l3redirect_port) {
    > +        if (op->od->n_l3dgw_ports) {
    > +            struct ovn_l3dgw_port *l3dgw_port =
    > +                ovn_get_l3dgw_port_from_lrp(op);
    >              ds_put_format(&match, "is_chassis_resident(%s)",
    > -                          op->od->l3redirect_port->json_key);
    > +                          l3dgw_port->redirect_port->json_key);
    >          }
    >      }
    >
    > @@ -9270,6 +9387,8 @@ build_adm_ctrl_flows_for_lrouter_port(
    >          struct ovn_port *op, struct hmap *lflows,
    >          struct ds *match, struct ds *actions)
    >  {
    > +    struct ovn_l3dgw_port *l3dgw_port = NULL;
    > +
    >      if (op->nbrp) {
    >          if (!lrport_is_enabled(op->nbrp)) {
    >              /* Drop packets from disabled logical ports (since logical flow
    > @@ -9300,12 +9419,12 @@ build_adm_ctrl_flows_for_lrouter_port(
    >          ds_clear(match);
    >          ds_put_format(match, "eth.dst == %s && inport == %s",
    >                        op->lrp_networks.ea_s, op->json_key);
    > -        if (op->od->l3dgw_port && op == op->od->l3dgw_port
    > -            && op->od->l3redirect_port) {
    > +        l3dgw_port = ovn_get_l3dgw_port_from_lrp(op);
    > +        if (l3dgw_port) {
    >              /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s
    >               * should only be received on the gateway chassis. */
    >              ds_put_format(match, " && is_chassis_resident(%s)",
    > -                          op->od->l3redirect_port->json_key);
    > +                          l3dgw_port->redirect_port->json_key);
    >          }
    >          ovn_lflow_add_with_hint(lflows, op->od, S_ROUTER_IN_ADMISSION, 50,
    >                                  ds_cstr(match),  ds_cstr(actions),
    > @@ -9425,6 +9544,8 @@ build_neigh_learning_flows_for_lrouter_port(
    >
    >          /* Check if we need to learn mac-binding from ARP requests. */
    >          for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
    > +            struct ovn_l3dgw_port *l3dgw_port =
    > +                ovn_get_l3dgw_port_from_lrp(op);
    >              if (!learn_from_arp_request) {
    >                  /* ARP request to this address should always get learned,
    >                   * so add a priority-110 flow to set
    > @@ -9437,10 +9558,9 @@ build_neigh_learning_flows_for_lrouter_port(
    >                                op->lrp_networks.ipv4_addrs[i].network_s,
    >                                op->lrp_networks.ipv4_addrs[i].plen,
    >                                op->lrp_networks.ipv4_addrs[i].addr_s);
    > -                if (op->od->l3dgw_port && op == op->od->l3dgw_port
    > -                    && op->od->l3redirect_port) {
    > +                if (l3dgw_port) {
    >                      ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                  op->od->l3redirect_port->json_key);
    > +                                  l3dgw_port->redirect_port->json_key);
    >                  }
    >                  const char *actions_s = REGBIT_LOOKUP_NEIGHBOR_RESULT
    >                                    " = lookup_arp(inport, arp.spa, arp.sha); "
    > @@ -9457,10 +9577,9 @@ build_neigh_learning_flows_for_lrouter_port(
    >                            op->json_key,
    >                            op->lrp_networks.ipv4_addrs[i].network_s,
    >                            op->lrp_networks.ipv4_addrs[i].plen);
    > -            if (op->od->l3dgw_port && op == op->od->l3dgw_port
    > -                && op->od->l3redirect_port) {
    > +            if (l3dgw_port) {
    >                  ds_put_format(match, " && is_chassis_resident(%s)",
    > -                              op->od->l3redirect_port->json_key);
    > +                              l3dgw_port->redirect_port->json_key);
    >              }
    >              ds_clear(actions);
    >              ds_put_format(actions, REGBIT_LOOKUP_NEIGHBOR_RESULT
    > @@ -9885,7 +10004,10 @@ build_arp_resolve_flows_for_lrouter_port(
    >              }
    >          }
    >
    > -        if (!op->derived && op->od->l3redirect_port) {
    > +        struct ovn_l3dgw_port *l3dgw_port =
    > +            ovn_get_l3dgw_port_from_lrp(op);
    > +
    > +        if (!op->derived && l3dgw_port) {
    >              const char *redirect_type = smap_get(&op->nbrp->options,
    >                                                   "redirect-type");
    >              if (redirect_type && !strcasecmp(redirect_type, "bridged")) {
    > @@ -9898,7 +10020,7 @@ build_arp_resolve_flows_for_lrouter_port(
    >                  ds_clear(match);
    >                  ds_put_format(match, "outport == %s && "
    >                                "!is_chassis_resident(%s)", op->json_key,
    > -                              op->od->l3redirect_port->json_key);
    > +                              l3dgw_port->redirect_port->json_key);
    >                  ds_clear(actions);
    >                  ds_put_format(actions, "eth.dst = %s; next;",
    >                                op->lrp_networks.ea_s);
    > @@ -10209,11 +10331,13 @@ build_check_pkt_len_flows_for_lrouter(
    >          ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1",
    >                        "next;");
    >
    > -        if (od->l3dgw_port && od->l3redirect_port) {
    > +        for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
    >              int gw_mtu = 0;
    > -            if (od->l3dgw_port->nbrp) {
    > -                 gw_mtu = smap_get_int(&od->l3dgw_port->nbrp->options,
    > -                                       "gateway_mtu", 0);
    > +            struct ovn_l3dgw_port *l3dgw_port = &od->l3dgw_ports[i];
    > +
    > +            if (l3dgw_port->dgw_port->nbrp) {
    > +                gw_mtu = smap_get_int(&(l3dgw_port->dgw_port->nbrp->options),
    > +                                      "gateway_mtu", 0);
    >              }
    >              /* Add the flows only if gateway_mtu is configured. */
    >              if (gw_mtu <= 0) {
    > @@ -10221,20 +10345,20 @@ build_check_pkt_len_flows_for_lrouter(
    >              }
    >
    >              ds_clear(match);
    > -            ds_put_format(match, "outport == %s", od->l3dgw_port->json_key);
    > -
    > +            ds_put_format(match, "outport == %s",
    > +                          l3dgw_port->dgw_port->json_key);
    >              ds_clear(actions);
    >              ds_put_format(actions,
    >                            REGBIT_PKT_LARGER" = check_pkt_larger(%d);"
    >                            " next;", gw_mtu + VLAN_ETH_HEADER_LEN);
    >              ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 50,
    >                                      ds_cstr(match), ds_cstr(actions),
    > -                                    &od->l3dgw_port->nbrp->header_);
    > +                                    &l3dgw_port->dgw_port->nbrp->header_);
    >
    > -            for (size_t i = 0; i < od->nbr->n_ports; i++) {
    > +            for (size_t j = 0; j < od->nbr->n_ports; j++) {
    >                  struct ovn_port *rp = ovn_port_find(ports,
    > -                                                    od->nbr->ports[i]->name);
    > -                if (!rp || rp == od->l3dgw_port) {
    > +                                                    od->nbr->ports[j]->name);
    > +                if (rp == l3dgw_port->dgw_port) {
    >                      continue;
    >                  }
    >
    > @@ -10242,7 +10366,8 @@ build_check_pkt_len_flows_for_lrouter(
    >                      ds_clear(match);
    >                      ds_put_format(match, "inport == %s && outport == %s"
    >                                    " && ip4 && "REGBIT_PKT_LARGER,
    > -                                  rp->json_key, od->l3dgw_port->json_key);
    > +                                  rp->json_key,
    > +                                  l3dgw_port->dgw_port->json_key);
    >
    >                      ds_clear(actions);
    >                      /* Set icmp4.frag_mtu to gw_mtu */
    > @@ -10271,7 +10396,8 @@ build_check_pkt_len_flows_for_lrouter(
    >                      ds_clear(match);
    >                      ds_put_format(match, "inport == %s && outport == %s"
    >                                    " && ip6 && "REGBIT_PKT_LARGER,
    > -                                  rp->json_key, od->l3dgw_port->json_key);
    > +                                  rp->json_key,
    > +                                  l3dgw_port->dgw_port->json_key);
    >
    >                      ds_clear(actions);
    >                      /* Set icmp6.frag_mtu to gw_mtu */
    > @@ -10313,11 +10439,13 @@ build_gateway_redirect_flows_for_lrouter(
    >          struct ds *match, struct ds *actions)
    >  {
    >      if (od->nbr) {
    > -        if (od->l3dgw_port && od->l3redirect_port) {
    > +        for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
    >              const struct ovsdb_idl_row *stage_hint = NULL;
    >
    > -            if (od->l3dgw_port->nbrp) {
    > -                stage_hint = &od->l3dgw_port->nbrp->header_;
    > +            struct ovn_l3dgw_port *l3dgw_port = &od->l3dgw_ports[i];
    > +
    > +            if (l3dgw_port->dgw_port->nbrp) {
    > +                stage_hint = &l3dgw_port->dgw_port->nbrp->header_;
    >              }
    >
    >              /* For traffic with outport == l3dgw_port, if the
    > @@ -10326,12 +10454,12 @@ build_gateway_redirect_flows_for_lrouter(
    >               * instance of the l3dgw_port. */
    >              ds_clear(match);
    >              ds_put_format(match, "outport == %s",
    > -                          od->l3dgw_port->json_key);
    > +                l3dgw_port->dgw_port->json_key);
    >              ds_clear(actions);
    >              ds_put_format(actions, "outport = %s; next;",
    > -                          od->l3redirect_port->json_key);
    > -            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50,
    > -                                    ds_cstr(match), ds_cstr(actions),
    > +                          l3dgw_port->redirect_port->json_key);
    > +            ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT,
    > +                                    50, ds_cstr(match), ds_cstr(actions),
    >                                      stage_hint);
    >          }
    >
    > @@ -10562,16 +10690,17 @@ build_ipv6_input_flows_for_lrouter_port(
    >          /* ND reply.  These flows reply to ND solicitations for the
    >           * router's own IP address. */
    >          for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
    > +            struct ovn_l3dgw_port *l3dgw_port =
    > +                ovn_get_l3dgw_port_from_lrp(op);
    >              ds_clear(match);
    > -            if (op->od->l3dgw_port && op == op->od->l3dgw_port
    > -                && op->od->l3redirect_port) {
    > +            if (l3dgw_port) {
    >                  /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
    >                   * should only be sent from the gateway chassi, so that
    >                   * upstream MAC learning points to the gateway chassis.
    >                   * Also need to avoid generation of multiple ND replies
    >                   * from different chassis. */
    >                  ds_put_format(match, "is_chassis_resident(%s)",
    > -                              op->od->l3redirect_port->json_key);
    > +                              l3dgw_port->redirect_port->json_key);
    >              }
    >
    >              build_lrouter_nd_flow(op->od, op, "nd_na_router",
    > @@ -10583,7 +10712,7 @@ build_ipv6_input_flows_for_lrouter_port(
    >
    >          /* UDP/TCP/SCTP port unreachable */
    >          if (!smap_get(&op->od->nbr->options, "chassis")
    > -            && !op->od->l3dgw_port) {
    > +            && !op->od->n_l3dgw_ports) {
    >              for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
    >                  ds_clear(match);
    >                  ds_put_format(match,
    > @@ -10799,17 +10928,19 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
    >                            op->lrp_networks.ipv4_addrs[i].network_s,
    >                            op->lrp_networks.ipv4_addrs[i].plen);
    >
    > -            if (op->od->l3dgw_port && op->od->l3redirect_port && op->peer
    > +            if (op->od->n_l3dgw_ports && op->peer
    >                  && op->peer->od->n_localnet_ports) {
    >                  bool add_chassis_resident_check = false;
    > -                if (op == op->od->l3dgw_port) {
    > +                struct ovn_l3dgw_port *l3dgw_port =
    > +                    ovn_get_l3dgw_port_from_lrp(op);
    > +                if (l3dgw_port) {
    >                      /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
    >                       * should only be sent from the gateway chassis, so that
    >                       * upstream MAC learning points to the gateway chassis.
    >                       * Also need to avoid generation of multiple ARP responses
    >                       * from different chassis. */
    >                      add_chassis_resident_check = true;
    > -                } else {
    > +                } else if (op->od->n_l3dgw_ports == 1) {
    >                      /* Check if the option 'reside-on-redirect-chassis'
    >                       * is set to true on the router port. If set to true
    >                       * and if peer's logical switch has a localnet port, it
    > @@ -10817,7 +10948,10 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
    >                       * peer's logical switch is be run on the chassis
    >                       * hosting the gateway port and it should reply to the
    >                       * ARP requests for the router port IPs.
    > +                     * 'reside-on-redirect-chassis' is supported only for
    > +                     * logical routers with single l3dgw port.
    >                       */
    > +                    l3dgw_port = &op->od->l3dgw_ports[0];
    >                      add_chassis_resident_check = smap_get_bool(
    >                          &op->nbrp->options,
    >                          "reside-on-redirect-chassis", false);
    > @@ -10825,7 +10959,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
    >
    >                  if (add_chassis_resident_check) {
    >                      ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                  op->od->l3redirect_port->json_key);
    > +                                  l3dgw_port->redirect_port->json_key);
    >                  }
    >              }
    >
    > @@ -10840,12 +10974,13 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
    >          struct sset all_ips_v6 = SSET_INITIALIZER(&all_ips_v6);
    >          get_router_load_balancer_ips(op->od, &all_ips_v4, &all_ips_v6);
    >
    > +        struct ovn_l3dgw_port *l3dgw_port = ovn_get_l3dgw_port_from_lrp(op);
    >          const char *ip_address;
    >          SSET_FOR_EACH (ip_address, &all_ips_v4) {
    >              ds_clear(match);
    > -            if (op == op->od->l3dgw_port) {
    > +            if (l3dgw_port) {
    >                  ds_put_format(match, "is_chassis_resident(%s)",
    > -                              op->od->l3redirect_port->json_key);
    > +                              l3dgw_port->redirect_port->json_key);
    >              }
    >
    >              build_lrouter_arp_flow(op->od, op,
    > @@ -10855,9 +10990,9 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
    >
    >          SSET_FOR_EACH (ip_address, &all_ips_v6) {
    >              ds_clear(match);
    > -            if (op == op->od->l3dgw_port) {
    > +            if (l3dgw_port) {
    >                  ds_put_format(match, "is_chassis_resident(%s)",
    > -                              op->od->l3redirect_port->json_key);
    > +                              l3dgw_port->redirect_port->json_key);
    >              }
    >
    >              build_lrouter_nd_flow(op->od, op, "nd_na",
    > @@ -10869,7 +11004,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
    >          sset_destroy(&all_ips_v6);
    >
    >          if (!smap_get(&op->od->nbr->options, "chassis")
    > -            && !op->od->l3dgw_port) {
    > +            && !op->od->n_l3dgw_ports) {
    >              /* UDP/TCP/SCTP port unreachable. */
    >              for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
    >                  ds_clear(match);
    > @@ -10946,7 +11081,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
    >           * exception is on the l3dgw_port where we might need to use a
    >           * different ETH address.
    >           */
    > -        if (op != op->od->l3dgw_port) {
    > +        if (!l3dgw_port) {
    >              return;
    >          }
    >
    > @@ -11011,7 +11146,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >          /* NAT rules are only valid on Gateway routers and routers with
    >           * l3dgw_port (router has a port with gateway chassis
    >           * specified). */
    > -        if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) {
    > +        if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) {
    >              return;
    >          }
    >
    > @@ -11027,8 +11162,9 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >
    >              nat = od->nbr->nat[i];
    >
    > -            ovs_be32 ip, mask;
    > +            ovs_be32 ip, mask, ip_external;
    >              struct in6_addr ipv6, mask_v6, v6_exact = IN6ADDR_EXACT_INIT;
    > +            struct in6_addr ip6_external;
    >              bool is_v6 = false;
    >              bool stateless = lrouter_nat_is_stateless(nat);
    >              struct nbrec_address_set *allowed_ext_ips =
    > @@ -11044,10 +11180,10 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                  continue;
    >              }
    >
    > -            char *error = ip_parse_masked(nat->external_ip, &ip, &mask);
    > +            char *error = ip_parse_masked(nat->external_ip, &ip_external, &mask);
    >              if (error || mask != OVS_BE32_MAX) {
    >                  free(error);
    > -                error = ipv6_parse_masked(nat->external_ip, &ipv6, &mask_v6);
    > +                error = ipv6_parse_masked(nat->external_ip, &ip6_external, &mask_v6);
    >                  if (error || memcmp(&mask_v6, &v6_exact, sizeof(mask_v6))) {
    >                      /* Invalid for both IPv4 and IPv6 */
    >                      static struct vlog_rate_limit rl =
    > @@ -11098,11 +11234,28 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                  }
    >              }
    >
    > +            struct ovn_l3dgw_port *l3dgw_port = NULL;
    > +            if (od->n_l3dgw_ports) {
    > +                /* Get the L3DGW port only for distributed router. */
    > +                if (!is_v6) {
    > +                    in6_addr_set_mapped_ipv4(&ip6_external, ip_external);
    > +                }
    > +                l3dgw_port = ovn_get_l3dgw_port_from_ip(od, ip6_external);
    > +                if (!l3dgw_port) {
    > +                    static struct vlog_rate_limit rl =
    > +                        VLOG_RATE_LIMIT_INIT(5, 1);
    > +                    VLOG_WARN_RL(&rl, "Could not map external ip: %s to a "
    > +                        "gateway port in router "UUID_FMT"",
    > +                        nat->external_ip, UUID_ARGS(&od->key));
    > +                    continue;
    > +                }
    > +            }
    > +
    >              /* For distributed router NAT, determine whether this NAT rule
    >               * satisfies the conditions for distributed NAT processing. */
    >              bool distributed = false;
    >              struct eth_addr mac;
    > -            if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") &&
    > +            if (l3dgw_port && !strcmp(nat->type, "dnat_and_snat") &&
    >                  nat->logical_port && nat->external_mac) {
    >                  if (eth_addr_from_string(nat->external_mac, &mac)) {
    >                      distributed = true;
    > @@ -11126,7 +11279,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >               * egress pipeline. */
    >              if (!strcmp(nat->type, "snat")
    >                  || !strcmp(nat->type, "dnat_and_snat")) {
    > -                if (!od->l3dgw_port) {
    > +                if (!l3dgw_port) {
    >                      /* Gateway router. */
    >                      ds_clear(match);
    >                      ds_clear(actions);
    > @@ -11154,12 +11307,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                                            " && inport == %s",
    >                                    is_v6 ? "6" : "4",
    >                                    nat->external_ip,
    > -                                  od->l3dgw_port->json_key);
    > -                    if (!distributed && od->l3redirect_port) {
    > +                                  l3dgw_port->dgw_port->json_key);
    > +                    if (!distributed && od->n_l3dgw_ports) {
    >                          /* Flows for NAT rules that are centralized are only
    >                           * programmed on the gateway chassis. */
    >                          ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                      od->l3redirect_port->json_key);
    > +                                      l3dgw_port->redirect_port->json_key);
    >                      }
    >
    >                      if (!strcmp(nat->type, "dnat_and_snat") && stateless) {
    > @@ -11181,7 +11334,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >               * to a logical IP address. */
    >              if (!strcmp(nat->type, "dnat")
    >                  || !strcmp(nat->type, "dnat_and_snat")) {
    > -                if (!od->l3dgw_port) {
    > +                if (!l3dgw_port) {
    >                      /* Gateway router. */
    >                      /* Packet when it goes from the initiator to destination.
    >                       * We need to set flags.loopback because the router can
    > @@ -11231,12 +11384,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                                            " && inport == %s",
    >                                    is_v6 ? "6" : "4",
    >                                    nat->external_ip,
    > -                                  od->l3dgw_port->json_key);
    > -                    if (!distributed && od->l3redirect_port) {
    > +                                  l3dgw_port->dgw_port->json_key);
    > +                    if (!distributed && l3dgw_port) {
    >                          /* Flows for NAT rules that are centralized are only
    >                           * programmed on the gateway chassis. */
    >                          ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                      od->l3redirect_port->json_key);
    > +                                      l3dgw_port->redirect_port->json_key);
    >                      }
    >                      ds_clear(actions);
    >                      if (allowed_ext_ips || exempted_ext_ips) {
    > @@ -11263,12 +11416,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >              }
    >
    >              /* ARP resolve for NAT IPs. */
    > -            if (od->l3dgw_port) {
    > +            if (l3dgw_port) {
    >                  if (!strcmp(nat->type, "snat")) {
    >                      ds_clear(match);
    >                      ds_put_format(
    >                          match, "inport == %s && %s == %s",
    > -                        od->l3dgw_port->json_key,
    > +                        l3dgw_port->dgw_port->json_key,
    >                          is_v6 ? "ip6.src" : "ip4.src", nat->external_ip);
    >                      ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_INPUT,
    >                                              120, ds_cstr(match), "next;",
    > @@ -11279,14 +11432,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                      ds_clear(match);
    >                      ds_put_format(
    >                          match, "outport == %s && %s == %s",
    > -                        od->l3dgw_port->json_key,
    > +                        l3dgw_port->dgw_port->json_key,
    >                          is_v6 ? REG_NEXT_HOP_IPV6 : REG_NEXT_HOP_IPV4,
    >                          nat->external_ip);
    >                      ds_clear(actions);
    >                      ds_put_format(
    >                          actions, "eth.dst = %s; next;",
    >                          distributed ? nat->external_mac :
    > -                        od->l3dgw_port->lrp_networks.ea_s);
    > +                        l3dgw_port->dgw_port->lrp_networks.ea_s);
    >                      ovn_lflow_add_with_hint(lflows, od,
    >                                              S_ROUTER_IN_ARP_RESOLVE,
    >                                              100, ds_cstr(match),
    > @@ -11309,19 +11462,19 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >               * Note that this only applies for NAT on a distributed router.
    >               * Undo DNAT on a gateway router is done in the ingress DNAT
    >               * pipeline stage. */
    > -            if (od->l3dgw_port && (!strcmp(nat->type, "dnat")
    > +            if (l3dgw_port && (!strcmp(nat->type, "dnat")
    >                  || !strcmp(nat->type, "dnat_and_snat"))) {
    >                  ds_clear(match);
    >                  ds_put_format(match, "ip && ip%s.src == %s"
    >                                        " && outport == %s",
    >                                is_v6 ? "6" : "4",
    >                                nat->logical_ip,
    > -                              od->l3dgw_port->json_key);
    > -                if (!distributed && od->l3redirect_port) {
    > +                              l3dgw_port->dgw_port->json_key);
    > +                if (!distributed && od->n_l3dgw_ports) {
    >                      /* Flows for NAT rules that are centralized are only
    >                       * programmed on the gateway chassis. */
    >                      ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                  od->l3redirect_port->json_key);
    > +                                  l3dgw_port->redirect_port->json_key);
    >                  }
    >                  ds_clear(actions);
    >                  if (distributed) {
    > @@ -11346,7 +11499,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >               * address. */
    >              if (!strcmp(nat->type, "snat")
    >                  || !strcmp(nat->type, "dnat_and_snat")) {
    > -                if (!od->l3dgw_port) {
    > +                if (!l3dgw_port) {
    >                      /* Gateway router. */
    >                      ds_clear(match);
    >                      ds_put_format(match, "ip && ip%s.src == %s",
    > @@ -11389,13 +11542,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                                            " && outport == %s",
    >                                    is_v6 ? "6" : "4",
    >                                    nat->logical_ip,
    > -                                  od->l3dgw_port->json_key);
    > -                    if (!distributed && od->l3redirect_port) {
    > +                                  l3dgw_port->dgw_port->json_key);
    > +                    if (!distributed && od->n_l3dgw_ports) {
    >                          /* Flows for NAT rules that are centralized are only
    >                           * programmed on the gateway chassis. */
    >                          priority += 128;
    >                          ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                      od->l3redirect_port->json_key);
    > +                                      l3dgw_port->redirect_port->json_key);
    >                      }
    >                      ds_clear(actions);
    >
    > @@ -11444,14 +11597,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                   */
    >                  ds_clear(actions);
    >                  ds_put_format(actions, REG_INPORT_ETH_ADDR " = %s; next;",
    > -                              od->l3dgw_port->lrp_networks.ea_s);
    > +                              l3dgw_port->dgw_port->lrp_networks.ea_s);
    >
    >                  ds_clear(match);
    >                  ds_put_format(match,
    >                                "eth.dst == "ETH_ADDR_FMT" && inport == %s"
    >                                " && is_chassis_resident(\"%s\")",
    >                                ETH_ADDR_ARGS(mac),
    > -                              od->l3dgw_port->json_key,
    > +                              l3dgw_port->dgw_port->json_key,
    >                                nat->logical_port);
    >                  ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ADMISSION, 50,
    >                                          ds_cstr(match), ds_cstr(actions),
    > @@ -11474,7 +11627,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                                "ip%s.src == %s && outport == %s && "
    >                                "is_chassis_resident(\"%s\")",
    >                                is_v6 ? "6" : "4", nat->logical_ip,
    > -                              od->l3dgw_port->json_key, nat->logical_port);
    > +                              l3dgw_port->dgw_port->json_key,
    > +                              nat->logical_port);
    >                  ds_put_format(actions, "eth.src = %s; %s = %s; next;",
    >                                nat->external_mac,
    >                                is_v6 ? REG_SRC_IPV6 : REG_SRC_IPV4,
    > @@ -11489,16 +11643,16 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >               * gateway port have ip.dst matching a NAT external IP, then
    >               * loop a clone of the packet back to the beginning of the
    >               * ingress pipeline with inport = outport. */
    > -            if (od->l3dgw_port) {
    > +            if (l3dgw_port) {
    >                  /* Distributed router. */
    >                  ds_clear(match);
    >                  ds_put_format(match, "ip%s.dst == %s && outport == %s",
    >                                is_v6 ? "6" : "4",
    >                                nat->external_ip,
    > -                              od->l3dgw_port->json_key);
    > +                              l3dgw_port->dgw_port->json_key);
    >                  if (!distributed) {
    >                      ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                  od->l3redirect_port->json_key);
    > +                                  l3dgw_port->redirect_port->json_key);
    >                  } else {
    >                      ds_put_format(match, " && is_chassis_resident(\"%s\")",
    >                                    nat->logical_port);
    > @@ -11522,7 +11676,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >          }
    >
    >          /* Handle force SNAT options set in the gateway router. */
    > -        if (!od->l3dgw_port) {
    > +        if (!od->n_l3dgw_ports) {
    >              if (dnat_force_snat_ip) {
    >                  if (od->dnat_force_snat_addrs.n_ipv4_addrs) {
    >                      build_lrouter_force_snat_flows(lflows, od, "4",
    > @@ -11561,7 +11715,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >
    >          /* Load balancing and packet defrag are only valid on
    >           * Gateway routers or router with gateway port. */
    > -        if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) {
    > +        if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) {
    >              sset_destroy(&nat_entries);
    >              return;
    >          }
    > @@ -11631,11 +11785,6 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
    >                      prio = 120;
    >                  }
    >
    > -                if (od->l3redirect_port &&
    > -                    (lb_vip->n_backends || !lb_vip->empty_backend_rej)) {
    > -                    ds_put_format(match, " && is_chassis_resident(%s)",
    > -                                  od->l3redirect_port->json_key);
    > -                }
    >                  add_router_lb_flow(lflows, od, match, actions, prio,
    >                                     lb_force_snat_ip, lb_vip, proto,
    >                                     nb_lb, meter_groups, &nat_entries);

    I think you need to update ovn-northd.8.xml since you've modified
    the logical flows.

    > diff --git a/ovn-nb.xml b/ovn-nb.xml
    > index b0a4adffe..6a6403023 100644
    > --- a/ovn-nb.xml
    > +++ b/ovn-nb.xml
    > @@ -2381,8 +2381,8 @@
    >            </p>
    >
    >            <p>
    > -            OVN honors this option only if the logical router has a distributed
    > -            gateway port and if the LRP's peer switch has a
    > +            OVN honors this option only if the logical router has a single
    > +            distributed gateway port and if the LRP's peer switch has a
    >              <code>localnet</code> port.
    >            </p>
    >          </column>
    > diff --git a/tests/ovn-nbctl.at b/tests/ovn-nbctl.at
    > index 6d91aa4c5..86f55fae3 100644
    > --- a/tests/ovn-nbctl.at
    > +++ b/tests/ovn-nbctl.at
    > @@ -721,8 +721,44 @@ AT_CHECK([ovn-nbctl lr-nat-update-ext-ip lr0 snat 192.168.16 allowed_range], [1]
    >  [ovn-nbctl: 192.168.16: Invalid IP address or CIDR
    >  ])
    >
    > -AT_CHECK([ovn-nbctl lr-nat-del lr0])])
    > +AT_CHECK([ovn-nbctl lr-nat-del lr0])
    > +AT_CHECK([ovn-nbctl lrp-add lr0 lrp0 00:00:00:01:02:03 192.168.1.1/24])
    > +AT_CHECK([ovn-nbctl lrp-add lr0 lrp1 00:00:00:01:02:04 172.64.1.1/24])
    > +AT_CHECK([ovn-nbctl lrp-set-gateway-chassis lrp0 chassis1])
    > +AT_CHECK([ovn-nbctl lrp-set-gateway-chassis lrp1 chassis2])
    >
    > +AT_CHECK([ovn-nbctl lr-nat-add lr0 snat 192.168.1.10 20.0.0.10])
    > +AT_CHECK([ovn-nbctl lr-nat-add lr0 snat 172.64.1.10 20.0.0.10])
    > +AT_CHECK([ovn-nbctl lr-nat-add lr0 snat 192.168.1.20 20.0.0.10], [1], [],
    > +[ovn-nbctl: a NAT with this type (snat) and logical_ip (20.0.0.10) already exists
    > +])
    > +AT_CHECK([ovn-nbctl lr-nat-add lr0 dnat 172.64.1.20 20.0.0.20])
    > +AT_CHECK([ovn-nbctl lr-nat-add lr0 dnat 172.64.1.20 20.0.0.30], [1], [],
    > +[ovn-nbctl: a NAT with this type (dnat) and external_ip (172.64.1.20) already exists
    > +])
    > +AT_CHECK([ovn-nbctl lr-nat-list lr0], [0], [dnl
    > +TYPE             EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
    > +dnat             172.64.1.20                         20.0.0.20
    > +snat             172.64.1.10                         20.0.0.10
    > +snat             192.168.1.10                        20.0.0.10
    > +])
    > +AT_CHECK([ovn-nbctl lr-nat-del lr0 snat 20.0.0.10 172.64.1.10])

    For all the above AT_CHECKs, you can replace with "check".
    Eg.
    check ovn-nbctl lr-nat-del lr0


    > +AT_CHECK([ovn-nbctl lr-nat-list lr0], [0], [dnl
    > +TYPE             EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
    > +dnat             172.64.1.20                         20.0.0.20
    > +snat             192.168.1.10                        20.0.0.10
    > +])
    > +AT_CHECK([ovn-nbctl lr-nat-add lr0 snat 172.64.1.10 20.0.0.10])
    > +AT_CHECK([ovn-nbctl lr-nat-del lr0 snat 20.0.0.10])
    Same as above. Replace AT_CHECK with check.

    > +AT_CHECK([ovn-nbctl lr-nat-list lr0], [0], [dnl
    > +TYPE             EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
    > +dnat             172.64.1.20                         20.0.0.20
    > +])
    > +AT_CHECK([ovn-nbctl lr-nat-del lr0 snat 20.0.0.10], [1], [],
    > +[ovn-nbctl: no matching NAT with the type (snat) and logical_ip (20.0.0.10)
    > +])
    > +AT_CHECK([ovn-nbctl lr-nat-del lr0])
    > +])
    >  dnl ---------------------------------------------------------------------
    >
    >  OVN_NBCTL_TEST([ovn_nbctl_lbs], [LBs], [
    > diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
    > index f3f88fa12..4d79104bf 100644
    > --- a/tests/ovn-northd.at
    > +++ b/tests/ovn-northd.at
    > @@ -836,7 +836,7 @@ ovn_start
    >  ovn-sbctl chassis-add gw1 geneve 127.0.0.1
    >
    >  ovn-nbctl lr-add R1
    > -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24
    > +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64
    >
    >  ovn-nbctl ls-add S1
    >  ovn-nbctl lsp-add S1 S1-R1
    > @@ -877,13 +877,13 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat  172.16.1.1
    >
    >  echo
    >  echo "IPv6: stateful"
    > -ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
    > +ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat 3000::c 1000::3
    >  check_flow_match_sets 2 2 2 0 0 0 0
    > -ovn-nbctl lr-nat-del R1 dnat_and_snat  fd01::1
    > +ovn-nbctl lr-nat-del R1 dnat_and_snat  3000::c
    >
    >  echo
    >  echo "IPv6: stateless"
    > -ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
    > +ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat 3000::c 1000::3

    You can use "check" for all the above ovn-nbctl commands and also for
    the rest of the places where you use ovn-nbctl/ovn-sbct/ovs-vsctl.

    Thanks
    Numan

    >  check_flow_match_sets 2 0 0 0 0 2 2
    >
    >  AT_CLEANUP
    > @@ -896,7 +896,7 @@ ovn_start
    >  ovn-sbctl chassis-add gw1 geneve 127.0.0.1
    >
    >  ovn-nbctl lr-add R1
    > -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24
    > +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64
    >
    >  ovn-nbctl ls-add S1
    >  ovn-nbctl lsp-add S1 S1-R1
    > @@ -1399,6 +1399,117 @@ OVS_WAIT_FOR_OUTPUT(
    >  AT_CLEANUP
    >  ])
    >
    > +AT_SETUP([ovn -- check Load balancer health check and Service Monitor sync for NAT])
    > +ovn_start
    > +
    > +ovn-nbctl lr-add lr0
    > +ovn-nbctl lrp-add lr0 lr0-public 00:00:01:01:02:04 192.168.2.1/24
    > +ovn-nbctl lrp-set-gateway-chassis lr0-public chassis1
    > +ovn-nbctl ls-add sw0
    > +ovn-nbctl ls-add sw1
    > +ovn-nbctl lrp-add lr0 lr0-sw0 00:00:00:00:00:05 10.0.0.1/24
    > +ovn-nbctl lrp-add lr0 lr0-sw1 02:00:00:00:00:05 20.0.0.1/24
    > +ovn-nbctl lsp-add sw0 sw0-lr0 -- set Logical_Switch_Port sw0-lr0 type=router options:router-port=lr0-sw0 -- lsp-set-addresses sw0-lr0 router
    > +ovn-nbctl lsp-add sw1 sw1-lr0 -- set Logical_Switch_Port sw1-lr0 type=router options:router-port=lr0-sw1 -- lsp-set-addresses sw1-lr0 router
    > +ovn-nbctl lr-nat-add lr0 snat 192.168.2.1 10.0.0.0/24
    > +
    > +ovn-nbctl --reject lb-add lb1 192.168.2.1:80 10.0.0.3:80,20.0.0.3:80
    > +
    > +AT_CHECK([ovn-nbctl --wait=sb -- --id=@hc create \
    > +Load_Balancer_Health_Check vip="192.168.2.1\:80" -- add Load_Balancer . \
    > +health_check @hc | uuidfilt], [0], [<0>
    > +])
    > +
    > +wait_row_count Service_Monitor 0
    > +
    > +ovn-nbctl --wait=sb lsp-add sw0 sw0-p1 -- lsp-set-addresses sw0-p1 \
    > +"00:00:00:00:00:03 10.0.0.3"
    > +ovn-nbctl --wait=sb lsp-add sw1 sw1-p1 -- lsp-set-addresses sw1-p1 \
    > +"02:00:00:00:00:03 20.0.0.3"
    > +
    > +ovn-nbctl --wait=sb set load_balancer . ip_port_mappings:10.0.0.3=sw0-p1:10.0.0.2
    > +wait_row_count Service_Monitor 1
    > +
    > +ovn-nbctl --wait=sb set load_balancer . ip_port_mappings:20.0.0.3=sw1-p1:20.0.0.2
    > +wait_row_count Service_Monitor 2
    > +
    > +check ovn-nbctl --wait=sb lr-lb-add lr0 lb1
    > +
    > +AT_CAPTURE_FILE([sbflows])
    > +OVS_WAIT_FOR_OUTPUT(
    > +  [ovn-sbctl dump-flows lr0 | tee sbflows | grep 'priority=120.*ct_lb' | sed 's/table=..//'], 0, [dnl
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80);)
    > +])
    > +
    > +# Get the uuid of both the service_monitor
    > +sm_sw0_p1=$(fetch_column Service_Monitor _uuid logical_port=sw0-p1)
    > +sm_sw1_p1=$(fetch_column Service_Monitor _uuid logical_port=sw1-p1)
    > +
    > +# Set the service monitor for sw1-p1 to offline
    > +check ovn-sbctl set service_monitor sw1-p1 status=offline
    > +wait_row_count Service_Monitor 1 logical_port=sw1-p1 status=offline
    > +
    > +AT_CAPTURE_FILE([sbflows2])
    > +OVS_WAIT_FOR_OUTPUT(
    > +  [ovn-sbctl dump-flows lr0 | tee sbflows2 | grep 'priority=120.*ct_lb' | sed 's/table=..//'], 0, [dnl
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80);)
    > +])
    > +
    > +# Set the service monitor for sw0-p1 to offline
    > +ovn-sbctl set service_monitor $sm_sw0_p1 status=offline
    > +
    > +wait_row_count Service_Monitor 1 logical_port=sw0-p1 status=offline
    > +
    > +AT_CAPTURE_FILE([sbflows3])
    > +OVS_WAIT_FOR_OUTPUT(
    > +  [ovn-sbctl dump-flows lr0 | tee sbflows3 | grep 'ct.new && ip && ip4.dst == 192.168.2.1' | grep 'priority=120' | sed 's/table=..//'], [0], [dnl
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(reg0 = 0; reject { outport <-> inport; next(pipeline=egress,table=1);};)
    > +])
    > +
    > +# Set the service monitor for sw0-p1 and sw1-p1 to online
    > +ovn-sbctl set service_monitor $sm_sw0_p1 status=online
    > +ovn-sbctl set service_monitor $sm_sw1_p1 status=online
    > +
    > +wait_row_count Service_Monitor 1 logical_port=sw1-p1 status=online
    > +
    > +# Add another distributed gateway port
    > +ovn-nbctl lrp-add lr0 lr0-public1 00:00:01:01:02:06 172.64.1.1/24
    > +ovn-nbctl lrp-set-gateway-chassis lr0-public1 chassis2
    > +
    > +AT_CAPTURE_FILE([sbflows4])
    > +OVS_WAIT_FOR_OUTPUT(
    > +  [ovn-sbctl dump-flows lr0 | tee sbflows4 | grep 'priority=120.*ct_lb' | sed 's/table=..//'], 0, [dnl
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80);)
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public1" && is_chassis_resident("cr-lr0-public1")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80);)
    > +])
    > +
    > +# Set the service monitors offline
    > +ovn-sbctl set service_monitor $sm_sw0_p1 status=offline
    > +ovn-sbctl set service_monitor $sm_sw1_p1 status=offline
    > +
    > +AT_CAPTURE_FILE([sbflows5])
    > +OVS_WAIT_FOR_OUTPUT(
    > +  [ovn-sbctl dump-flows lr0 | tee sbflows5 | grep 'ct.new && ip && ip4.dst == 192.168.2.1' | grep 'priority=120' | sed 's/table=..//'], [0], [dnl
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(reg0 = 0; reject { outport <-> inport; next(pipeline=egress,table=1);};)
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public1" && is_chassis_resident("cr-lr0-public1")), action=(reg0 = 0; reject { outport <-> inport; next(pipeline=egress,table=1);};)
    > +])
    > +
    > +# Set reject=false on lb1
    > +ovn-nbctl set Load_Balancer lb1 options:reject=false
    > +
    > +AT_CAPTURE_FILE([sbflows6])
    > +OVS_WAIT_FOR_OUTPUT(
    > +  [ovn-sbctl dump-flows lr0 | tee sbflows6 | grep 'priority=120.*ct_lb'], 1)
    > +
    > +AT_CAPTURE_FILE([sbflows7])
    > +OVS_WAIT_FOR_OUTPUT(
    > +  [ovn-sbctl dump-flows lr0 | tee sbflows7 | grep 'ct.new && ip && ip4.dst == 192.168.2.1' | grep priority=120 | sed 's/table=..//'], [0], [dnl
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(drop;)
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 192.168.2.1 && tcp && tcp.dst == 80 && inport == "lr0-public1" && is_chassis_resident("cr-lr0-public1")), action=(drop;)
    > +])
    > +
    > +AT_CLEANUP
    > +
    >  OVN_FOR_EACH_NORTHD([
    >  AT_SETUP([ovn -- Load balancer VIP in NAT entries])
    >  AT_SKIP_IF([test $HAVE_PYTHON = no])
    > @@ -2674,6 +2785,348 @@ check ovn-nbctl ls-add ls \
    >  check ovn-nbctl --wait=sb sync
    >
    >  AT_CHECK([grep -qE 'duplicate logical.*port p1' northd/ovn-northd.log], [0])
    > +AT_CLEANUP
    > +
    > +AT_SETUP([ovn-northd -- lr multiple gw ports])
    > +ovn_start
    > +
    > +# Logical network:
    > +# 1 LR, 3 Logical Switches,
    > +# 1 gateway chassis attached to each corresponding LRP.
    > +#
    > +#                | S1 (gw1)
    > +#                |
    > +#      ls  ----  DR -- S3 (gw3)
    > +# (20.0.0.0/24)  |
    > +#                | S2 (gw2)
    > +#
    > +# We will validate basic LR logical flows.
    > +
    > +ovn-sbctl chassis-add gw1 geneve 127.0.0.1
    > +ovn-sbctl chassis-add gw2 geneve 128.0.0.1
    > +ovn-sbctl chassis-add gw3 geneve 129.0.0.1
    > +
    > +ovn-nbctl lr-add DR
    > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
    > +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.0/24
    > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.0/24
    > +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24
    > +
    > +ovn-nbctl ls-add S1
    > +ovn-nbctl lsp-add S1 S1-DR
    > +ovn-nbctl lsp-set-type S1-DR router
    > +ovn-nbctl lsp-set-addresses S1-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
    > +
    > +ovn-nbctl ls-add S2
    > +ovn-nbctl lsp-add S2 S2-DR
    > +ovn-nbctl lsp-set-type S2-DR router
    > +ovn-nbctl lsp-set-addresses S2-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
    > +
    > +ovn-nbctl ls-add S3
    > +ovn-nbctl lsp-add S3 S3-DR
    > +ovn-nbctl lsp-set-type S3-DR router
    > +ovn-nbctl lsp-set-addresses S3-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
    > +
    > +ovn-nbctl ls-add  ls
    > +ovn-nbctl lsp-add ls ls-DR
    > +ovn-nbctl lsp-set-type ls-DR router
    > +ovn-nbctl lsp-set-addresses ls-DR router
    > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
    > +
    > +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
    > +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
    > +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
    > +
    > +ovn-nbctl --wait=sb sync
    > +
    > +ovn-sbctl dump-flows DR
    > +
    > +# Check the flows in lr_in_lookup_neighbor stage
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR | wc -l], [0], [3
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S1 | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S2 | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S3 | wc -l], [0], [1
    > +])
    > +
    > +# Check the flows in lr_in_gw_redirect stage
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR | wc -l], [0], [3
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S1 | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S2 | wc -l], [0], [1
    > +])
    > +AT_CLEANUP
    > +
    > +AT_SETUP([ovn-northd -- lr multiple gw ports NAT])
    > +ovn_start
    > +
    > +# Logical network:
    > +# 1 LR, 3 Logical Switches,
    > +# 1 gateway chassis attached to each corresponding LRP.
    > +#
    > +#                | S1 (gw1)
    > +#                |
    > +#      ls  ----  DR -- S3 (gw3)
    > +# (20.0.0.0/24)  |
    > +#                | S2 (gw2)
    > +#
    > +# We will validate basic SNAT, DNAT and DNAT_AND_SNAT with
    > +# multiple distributed gateway LRPs.
    > +ovn-sbctl chassis-add gw1 geneve 127.0.0.1
    > +ovn-sbctl chassis-add gw2 geneve 128.0.0.1
    > +ovn-sbctl chassis-add gw3 geneve 129.0.0.1
    > +
    > +ovn-nbctl lr-add DR
    > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
    > +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24
    > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
    > +ovn-nbctl lrp-add DR DR-ls 05:ac:10:01:00:01 20.0.0.0/24
    > +
    > +ovn-nbctl ls-add S1
    > +ovn-nbctl lsp-add S1 S1-DR
    > +ovn-nbctl lsp-set-type S1-DR router
    > +ovn-nbctl lsp-set-addresses S1-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
    > +
    > +ovn-nbctl ls-add S2
    > +ovn-nbctl lsp-add S2 S2-DR
    > +ovn-nbctl lsp-set-type S2-DR router
    > +ovn-nbctl lsp-set-addresses S2-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
    > +
    > +ovn-nbctl ls-add S3
    > +ovn-nbctl lsp-add S3 S3-DR
    > +ovn-nbctl lsp-set-type S3-DR router
    > +ovn-nbctl lsp-set-addresses S3-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
    > +
    > +ovn-nbctl ls-add  ls
    > +ovn-nbctl lsp-add ls ls-DR
    > +ovn-nbctl lsp-set-type ls-DR router
    > +ovn-nbctl lsp-set-addresses ls-DR router
    > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
    > +
    > +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
    > +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
    > +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
    > +
    > +ovn-nbctl --wait=sb sync
    > +
    > +# Configure SNAT
    > +ovn-nbctl lr-nat-add DR snat  172.16.1.1  20.0.0.10
    > +ovn-nbctl lr-nat-add DR snat  10.0.0.1    20.0.0.10
    > +ovn-nbctl lr-nat-add DR snat  192.168.0.1 20.0.0.10
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.1" | grep  cr-DR-S1 | grep ct_snat | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.1)"| wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.1" | grep  cr-DR-S2 | grep ct_snat | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.1)"| wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.1" | grep  cr-DR-S3 | grep ct_snat | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.1)"| wc -l], [0], [1
    > +])
    > +
    > +ovn-nbctl lr-nat-del DR snat  20.0.0.10
    > +ovn-nbctl lr-nat-del DR snat  20.0.0.10
    > +ovn-nbctl lr-nat-del DR snat  20.0.0.10
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0
    > +])
    > +
    > +# Configure DNAT
    > +ovn-nbctl lr-nat-add DR dnat  172.16.1.10  20.0.0.10
    > +ovn-nbctl lr-nat-add DR dnat  10.0.0.10    20.0.0.10
    > +ovn-nbctl lr-nat-add DR dnat  192.168.0.10 20.0.0.10
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep  cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep  cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep  cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1
    > +])
    > +
    > +ovn-nbctl lr-nat-del DR dnat  172.16.1.10
    > +ovn-nbctl lr-nat-del DR dnat  10.0.0.10
    > +ovn-nbctl lr-nat-del DR dnat  192.168.0.10
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0
    > +])
    > +
    > +# Configure DNAT_AND_SNAT
    > +ovn-nbctl lr-nat-add DR dnat_and_snat  172.16.1.10    20.0.0.10
    > +ovn-nbctl lr-nat-add DR dnat_and_snat  10.0.0.10      20.0.0.10
    > +ovn-nbctl lr-nat-add DR dnat_and_snat  192.168.0.10   20.0.0.10
    > +
    > +ovn-sbctl dump-flows DR
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.10" | grep  cr-DR-S1 | grep ct_snat | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.10)"| wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.10" | grep  cr-DR-S2 | grep ct_snat | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.10)"| wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.10" | grep  cr-DR-S3 | grep ct_snat | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.10)"| wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep  cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep  cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep  cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1
    > +])
    > +
    > +ovn-nbctl lr-nat-del DR dnat_and_snat  172.16.1.10
    > +ovn-nbctl lr-nat-del DR dnat_and_snat  10.0.0.10
    > +ovn-nbctl lr-nat-del DR dnat_and_snat  192.168.0.10
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0
    > +])
    > +AT_CLEANUP
    > +
    > +AT_SETUP([ovn-northd -- lr multiple gw ports LB])
    > +ovn_start
    > +
    > +# Logical network:
    > +# 1 LR, 3 Logical Switches,
    > +# 1 gateway chassis attached to each corresponding LRP.
    > +#
    > +#                | S1 (gw1)
    > +#                |
    > +#      ls  ----  DR -- S3 (gw3)
    > +# (20.0.0.0/24)  |
    > +#                | S2 (gw2)
    > +#
    > +# We will validate LB with multiple distributed gateway LRPs.
    > +ovn-sbctl chassis-add gw1 geneve 127.0.0.1
    > +ovn-sbctl chassis-add gw2 geneve 128.0.0.1
    > +ovn-sbctl chassis-add gw3 geneve 129.0.0.1
    > +
    > +ovn-nbctl lr-add DR
    > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
    > +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24
    > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
    > +ovn-nbctl lrp-add DR DR-ls 05:ac:10:01:00:01 20.0.0.0/24
    > +
    > +ovn-nbctl ls-add S1
    > +ovn-nbctl lsp-add S1 S1-DR
    > +ovn-nbctl lsp-set-type S1-DR router
    > +ovn-nbctl lsp-set-addresses S1-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
    > +
    > +ovn-nbctl ls-add S2
    > +ovn-nbctl lsp-add S2 S2-DR
    > +ovn-nbctl lsp-set-type S2-DR router
    > +ovn-nbctl lsp-set-addresses S2-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
    > +
    > +ovn-nbctl ls-add S3
    > +ovn-nbctl lsp-add S3 S3-DR
    > +ovn-nbctl lsp-set-type S3-DR router
    > +ovn-nbctl lsp-set-addresses S3-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
    > +
    > +ovn-nbctl ls-add  ls
    > +ovn-nbctl lsp-add ls ls-DR
    > +ovn-nbctl lsp-set-type ls-DR router
    > +ovn-nbctl lsp-set-addresses ls-DR router
    > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
    > +
    > +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
    > +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
    > +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
    > +
    > +ovn-nbctl --wait=sb sync
    > +
    > +ovn-nbctl lb-add lb0 192.168.0.3:80 10.0.0.2:80,10.0.0.3:80
    > +ovn-nbctl lr-lb-add DR lb0
    > +
    > +ovn-sbctl dump-flows DR
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| wc -l], [0], [3
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| wc -l], [0], [3
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S1 | grep cr-DR-S1| wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S1| grep cr-DR-S1 | wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S2 | grep cr-DR-S2| wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S2| grep cr-DR-S2 | wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S3 | grep cr-DR-S3| wc -l], [0], [1
    > +])
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S3| grep cr-DR-S3 | wc -l], [0], [1
    > +])
    > +
    > +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep priority=120| wc -l], [0], [3
    > +])
    >
    >  AT_CLEANUP
    >
    > @@ -2878,4 +3331,4 @@ wait_row_count FDB 0
    >  ovn-sbctl list FDB
    >
    >  AT_CLEANUP
    > -])
    > \ No newline at end of file
    > +])
    > diff --git a/tests/ovn.at b/tests/ovn.at
    > index bec593dcc..4e1c0fc43 100644
    > --- a/tests/ovn.at
    > +++ b/tests/ovn.at
    > @@ -19943,7 +19943,7 @@ AT_CAPTURE_FILE([sbflows2])
    >  OVS_WAIT_FOR_OUTPUT(
    >    [ovn-sbctl dump-flows > sbflows2
    >     ovn-sbctl dump-flows lr0 | grep ct_lb | grep priority=120 | sed 's/table=..//'], 0,
    > -  [  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");)
    > +  [  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");)
    >  ])
    >
    >  # get the svc monitor mac.
    > @@ -19984,8 +19984,8 @@ AT_CHECK(
    >  AT_CAPTURE_FILE([sbflows4])
    >  ovn-sbctl dump-flows lr0 > sbflows4
    >  AT_CHECK([grep lr_in_dnat sbflows4 | grep priority=120 | sed 's/table=..//' | sort], [0], [dnl
    > -  (lr_in_dnat         ), priority=120  , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
    > -  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(drop;)
    > +  (lr_in_dnat         ), priority=120  , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
    > +  (lr_in_dnat         ), priority=120  , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(drop;)
    >  ])
    >
    >  # Delete sw0-p1
    > @@ -25149,3 +25149,307 @@ AT_CHECK([cat hv2_offlows_table72.txt | grep -v NXST], [1], [dnl
    >  OVN_CLEANUP([hv1], [hv2])
    >  AT_CLEANUP
    >  ])
    > +
    > +AT_SETUP([ovn -- lr multiple gw ports])
    > +ovn_start
    > +
    > +# Logical network:
    > +# 1 LR, 3 Logical Switches,
    > +# 1 gateway chassis attached to each corresponding LRP.
    > +#
    > +#                | S1 (gw1)
    > +#                |
    > +#      ls  ----  DR -- S3 (gw3)
    > +# (20.0.0.0/24)  |
    > +#                | S2 (gw2)
    > +#
    > +# S1 - VLAN 1000
    > +# S2 - VLAN 2000
    > +# S3 - VLAN 3000
    > +#
    > +# 5 chassis(s), HV1----HV5
    > +#
    > +# HV1 - VIF11
    > +# HV2 - Gateway chassis gw1
    > +# HV3 - Gateway chassis gw2
    > +# HV4 - Gateway chassis gw3
    > +# HV5 - North endpoint
    > +
    > +ovn-nbctl lr-add DR
    > +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
    > +ovn-nbctl lrp-add DR DR-S2 08:ac:10:01:00:01 10.0.0.1/24
    > +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
    > +ovn-nbctl lrp-add DR DR-ls 06:ac:10:01:00:01 20.0.0.0/24
    > +
    > +ovn-nbctl ls-add S1
    > +ovn-nbctl lsp-add S1 S1-DR
    > +ovn-nbctl lsp-set-type S1-DR router
    > +ovn-nbctl lsp-set-addresses S1-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
    > +ovn-nbctl lsp-add S1 ln1 "" 1000
    > +ovn-nbctl lsp-set-addresses ln1 unknown
    > +ovn-nbctl lsp-set-type ln1 localnet
    > +ovn-nbctl lsp-set-options ln1 network_name=phys
    > +
    > +ovn-nbctl ls-add S2
    > +ovn-nbctl lsp-add S2 S2-DR
    > +ovn-nbctl lsp-set-type S2-DR router
    > +ovn-nbctl lsp-set-addresses S2-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
    > +ovn-nbctl lsp-add S2 ln2 "" 2000
    > +ovn-nbctl lsp-set-addresses ln2 unknown
    > +ovn-nbctl lsp-set-type ln2 localnet
    > +ovn-nbctl lsp-set-options ln2 network_name=phys
    > +
    > +ovn-nbctl ls-add S3
    > +ovn-nbctl lsp-add S3 S3-DR
    > +ovn-nbctl lsp-set-type S3-DR router
    > +ovn-nbctl lsp-set-addresses S3-DR router
    > +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
    > +ovn-nbctl lsp-add S3 ln3 "" 3000
    > +ovn-nbctl lsp-set-addresses ln3 unknown
    > +ovn-nbctl lsp-set-type ln3 localnet
    > +ovn-nbctl lsp-set-options ln3 network_name=phys
    > +
    > +ovn-nbctl ls-add  ls
    > +ovn-nbctl lsp-add ls ls-DR
    > +ovn-nbctl lsp-set-type ls-DR router
    > +ovn-nbctl lsp-set-addresses ls-DR router
    > +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
    > +
    > +# Add the lsp lp11 to ls. This will map to VIF11.
    > +ovn-nbctl lsp-add ls lp11
    > +ovn-nbctl lsp-set-addresses lp11 "f0:00:00:00:00:10 20.0.0.10"
    > +ovn-nbctl lsp-set-port-security lp11 f0:00:00:00:00:10
    > +
    > +# Add the Northbound endpoint, lp-north1
    > +ovn-nbctl ls-add ls-north1
    > +ovn-nbctl lsp-add ls-north1 ln4 "" 1000
    > +ovn-nbctl lsp-set-addresses ln4 unknown
    > +ovn-nbctl lsp-set-type ln4 localnet
    > +ovn-nbctl lsp-set-options ln4 network_name=phys
    > +
    > +ovn-nbctl lsp-add ls-north1 lp-north1
    > +ovn-nbctl lsp-set-addresses lp-north1 "f0:f0:00:00:00:11 172.16.1.10"
    > +ovn-nbctl lsp-set-port-security lp-north1 f0:f0:00:00:00:11
    > +
    > +# Add the Northbound endpoint, lp-north2
    > +ovn-nbctl ls-add ls-north2
    > +ovn-nbctl lsp-add ls-north2 ln5 "" 2000
    > +ovn-nbctl lsp-set-addresses ln5 unknown
    > +ovn-nbctl lsp-set-type ln5 localnet
    > +ovn-nbctl lsp-set-options ln5 network_name=phys
    > +
    > +ovn-nbctl lsp-add ls-north2 lp-north2
    > +ovn-nbctl lsp-set-addresses lp-north2 "f0:f0:00:00:00:22 10.0.0.10"
    > +ovn-nbctl lsp-set-port-security lp-north2 f0:f0:00:00:00:22
    > +
    > +# Add the Northbound endpoint, lp-north3
    > +ovn-nbctl ls-add ls-north3
    > +ovn-nbctl lsp-add ls-north3 ln6 "" 3000
    > +ovn-nbctl lsp-set-addresses ln6 unknown
    > +ovn-nbctl lsp-set-type ln6 localnet
    > +ovn-nbctl lsp-set-options ln6 network_name=phys
    > +
    > +ovn-nbctl lsp-add ls-north3 lp-north3
    > +ovn-nbctl lsp-set-addresses lp-north3 "f0:f0:00:00:00:33 192.168.0.10"
    > +ovn-nbctl lsp-set-port-security lp-north3 f0:f0:00:00:00:33
    > +
    > +# Add 5 chassis
    > +net_add n1
    > +for i in 1 2 3 4 5; do
    > +    sim_add hv$i
    > +    as hv$i
    > +    ovs-vsctl add-br br-phys
    > +    ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
    > +    ovn_attach n1 br-phys 192.168.0.$i 24 $encap
    > +done
    > +
    > +# Add a vif on HV1
    > +as hv1 ovs-vsctl add-port br-int vif11 -- \
    > +    set Interface vif11 external-ids:iface-id=lp11 \
    > +                              options:tx_pcap=hv1/vif11-tx.pcap \
    > +                              options:rxq_pcap=hv1/vif11-rx.pcap \
    > +                              ofport-request=11
    > +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp11` = xup])
    > +
    > +as hv5 ovs-vsctl add-port br-int vif-north1 -- \
    > +        set Interface vif-north1 external-ids:iface-id=lp-north1 \
    > +                              options:tx_pcap=hv5/vif-north1-tx.pcap \
    > +                              options:rxq_pcap=hv5/vif-north1-rx.pcap \
    > +                              ofport-request=44
    > +
    > +as hv5 ovs-vsctl add-port br-int vif-north2 -- \
    > +        set Interface vif-north2 external-ids:iface-id=lp-north2 \
    > +                              options:tx_pcap=hv5/vif-north2-tx.pcap \
    > +                              options:rxq_pcap=hv5/vif-north2-rx.pcap \
    > +                              ofport-request=45
    > +
    > +as hv5 ovs-vsctl add-port br-int vif-north3 -- \
    > +        set Interface vif-north3 external-ids:iface-id=lp-north3 \
    > +                              options:tx_pcap=hv5/vif-north3-tx.pcap \
    > +                              options:rxq_pcap=hv5/vif-north3-rx.pcap \
    > +                              ofport-request=46
    > +
    > +ovn-nbctl lrp-set-gateway-chassis DR-S1 hv2
    > +ovn-nbctl lrp-set-gateway-chassis DR-S2 hv3
    > +ovn-nbctl lrp-set-gateway-chassis DR-S3 hv4
    > +
    > +ovn-nbctl --wait=sb sync
    > +OVN_POPULATE_ARP
    > +
    > +vif_to_ls () {
    > +    case ${1} in dnl (
    > +        vif?[[11]]) echo ls ;; dnl (
    > +        vif-north1) echo ls-north1 ;; dnl (
    > +        vif-north2) echo ls-north2 ;; dnl (
    > +        vif-north3) echo ls-north3 ;; dnl (
    > +        *) AT_FAIL_IF([:]) ;;
    > +    esac
    > +}
    > +
    > +vif_to_hv () {
    > +    case ${1} in dnl (
    > +        vif[[1]]?) echo hv1 ;; dnl (
    > +        vif-north1) echo hv5 ;; dnl (
    > +        vif-north2) echo hv5 ;; dnl (
    > +        vif-north3) echo hv5 ;; dnl (
    > +        *) AT_FAIL_IF([:]) ;;
    > +    esac
    > +}
    > +
    > +vif_to_lrp () {
    > +    case ${1} in dnl (
    > +        vif?[[11]]) echo DR-ls ;; dnl (
    > +        *) AT_FAIL_IF([:]) ;;
    > +    esac
    > +
    > +}
    > +
    > +ip_to_hex() {
    > +       printf "%02x%02x%02x%02x" "${@}"
    > +}
    > +
    > +# test_arp INPORT SHA SPA TPA
    > +#
    > +# Causes a packet to be received on INPORT.  The packet is an ARP
    > +# request with SHA, SPA, and TPA as specified.
    > +test_arp() {
    > +    local inport=$1 sha=$2 spa=$3 tpa=$4
    > +    local request=ffffffffffff${sha}08060001080006040001${sha}${spa}ffffffffffff${tpa}
    > +    hv=`vif_to_hv $inport`
    > +    as $hv ovs-appctl netdev-dummy/receive $inport $request
    > +}
    > +
    > +
    > +test_ip() {
    > +        # This packet has bad checksums but logical L3 routing doesn't check.
    > +        local inport=${1} src_mac=${2} dst_mac=${3} src_ip=${4} dst_ip=${5} outport=${6}
    > +        local packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000
    > +        shift; shift; shift; shift; shift
    > +        hv=`vif_to_hv $inport`
    > +        as $hv ovs-appctl netdev-dummy/receive $inport $packet
    > +        in_ls=`vif_to_ls $inport`
    > +        for outport; do
    > +            out_ls=`vif_to_ls $outport`
    > +            if test $in_ls = $out_ls; then
    > +                # Ports on the same logical switch receive exactly the same packet.
    > +                echo $packet
    > +            else
    > +                # Routing decrements TTL and updates source and dest MAC
    > +                # (and checksum).
    > +                # For North-South, packet will come via gateway chassis, i.e hv3
    > +                if test $inport = vif-north1; then
    > +                    echo f0000000001006ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
    > +                fi
    > +                if test $outport = vif-north1; then
    > +                    echo f0f00000001102ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
    > +                fi
    > +                if test $outport = vif-north2; then
    > +                    echo f0f00000002208ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
    > +                fi
    > +                if test $outport = vif-north3; then
    > +                    echo f0f00000003304ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
    > +                fi
    > +            fi >> $outport.expected
    > +        done
    > +}
    > +
    > +echo "------ OVN dump ------"
    > +ovn-nbctl show
    > +ovn-sbctl show
    > +ovn-sbctl list port_binding
    > +ovn-sbctl list mac_binding
    > +ovn-sbctl list datapath_binding
    > +
    > +ovn-sbctl dump-flows DR
    > +ovn-sbctl dump-flows S1
    > +ovn-sbctl dump-flows ls
    > +
    > +echo "------ hv1 dump ------"
    > +as hv1 ovs-vsctl show
    > +as hv1 ovs-vsctl list Open_Vswitch
    > +as hv1 ovs-ofctl dump-flows br-int
    > +
    > +echo "------ hv2 dump ------"
    > +as hv2 ovs-vsctl show
    > +as hv2 ovs-vsctl list Open_Vswitch
    > +as hv2 ovs-ofctl dump-flows br-int
    > +
    > +echo "------ hv3 dump ------"
    > +as hv3 ovs-vsctl show
    > +as hv3 ovs-vsctl list Open_Vswitch
    > +as hv3 ovs-ofctl dump-flows br-int
    > +
    > +echo "------ hv4 dump ------"
    > +as hv4 ovs-vsctl show
    > +as hv4 ovs-vsctl list Open_Vswitch
    > +as hv5 ovs-ofctl dump-flows br-int
    > +
    > +# N-S with lp-north1
    > +echo "Send Dummy ARP"
    > +sip=`ip_to_hex 172 16 1 10`
    > +tip=`ip_to_hex 172 16 1 50`
    > +test_arp vif-north1 f0f000000011 $sip $tip
    > +
    > +echo "Send traffic North to South"
    > +sip=`ip_to_hex 172 16 1 10`
    > +dip=`ip_to_hex 20 0 0 10`
    > +test_ip vif-north1 f0f000000011 02ac10010001 $sip $dip vif11
    > +# Confirm that North to south traffic works fine.
    > +OVN_CHECK_PACKETS([hv1/vif11-tx.pcap], [vif11.expected])
    > +
    > +echo "Send traffic South to North"
    > +sip=`ip_to_hex 20 0 0 10`
    > +dip=`ip_to_hex 172 16 1 10`
    > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north1
    > +# Confirm that South to North traffic works fine.
    > +OVN_CHECK_PACKETS([hv5/vif-north1-tx.pcap], [vif-north1.expected])
    > +
    > +# N-S with lp-north2
    > +echo "Send Dummy ARP"
    > +sip=`ip_to_hex 10 0 0 10`
    > +tip=`ip_to_hex 10 0 0 50`
    > +test_arp vif-north2 f0f000000022 $sip $tip
    > +
    > +echo "Send traffic South to North"
    > +sip=`ip_to_hex 20 0 0 10`
    > +dip=`ip_to_hex 10 0 0 10`
    > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north2
    > +# Confirm that South to North traffic works fine.
    > +OVN_CHECK_PACKETS([hv5/vif-north2-tx.pcap], [vif-north2.expected])
    > +
    > +# N-S with lp-north3
    > +echo "Send Dummy ARP"
    > +sip=`ip_to_hex 192 168 0 10`
    > +tip=`ip_to_hex 192 168 0 50`
    > +test_arp vif-north3 f0f000000033 $sip $tip
    > +
    > +echo "Send traffic South to North"
    > +sip=`ip_to_hex 20 0 0 10`
    > +dip=`ip_to_hex 192 168 0 10`
    > +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north3
    > +# Confirm that South to North traffic works fine.
    > +OVN_CHECK_PACKETS([hv5/vif-north3-tx.pcap], [vif-north3.expected])
    > +
    > +AT_CLEANUP
    > diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c
    > index 2c77f4ba7..af93622a5 100644
    > --- a/utilities/ovn-nbctl.c
    > +++ b/utilities/ovn-nbctl.c
    > @@ -781,7 +781,7 @@ NAT commands:\n\
    >    lr-nat-add ROUTER TYPE EXTERNAL_IP LOGICAL_IP [LOGICAL_PORT EXTERNAL_MAC]\n\
    >                              [EXTERNAL_PORT_RANGE]\n\
    >                              add a NAT to ROUTER\n\
    > -  lr-nat-del ROUTER [TYPE [IP]]\n\
    > +  lr-nat-del ROUTER [TYPE [IP [EXTERNAL_IP]]]\n\
    >                              remove NATs from ROUTER\n\
    >    lr-nat-list ROUTER        print NATs for ROUTER\n\
    >  \n\
    > @@ -4285,6 +4285,90 @@ done:
    >      return ret;
    >  }
    >
    > +static bool
    > +is_nat_rule_conflict(const struct nbrec_logical_router *lr,
    > +                     char *new_external_ip, char *old_external_ip, bool is_v6)
    > +{
    > +    int num_l3dgw_ports = 0;
    > +    bool is_conflict = false;
    > +
    > +    struct lport_addresses old_external_ip_addr, new_external_ip_addr;
    > +
    > +    if (!extract_ip_addresses(new_external_ip, &new_external_ip_addr) ||
    > +        !extract_ip_addresses(old_external_ip, &old_external_ip_addr)) {
    > +        static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
    > +        VLOG_WARN_RL(&rl, "Extract addresses failed.");
    > +        return true;
    > +    }
    > +
    > +    if (is_v6) {
    > +        ovs_assert(new_external_ip_addr.n_ipv6_addrs == 1);
    > +        ovs_assert(old_external_ip_addr.n_ipv6_addrs == 1);
    > +    } else {
    > +        ovs_assert(new_external_ip_addr.n_ipv4_addrs == 1);
    > +        ovs_assert(old_external_ip_addr.n_ipv4_addrs == 1);
    > +    }
    > +
    > +    for (size_t i = 0; i < lr->n_ports; i++) {
    > +        const struct nbrec_logical_router_port *lrp = lr->ports[i];
    > +        const struct nbrec_logical_router_port *old_port = NULL;
    > +        const struct nbrec_logical_router_port *new_port = NULL;
    > +        if (lrp->n_gateway_chassis) {
    > +            num_l3dgw_ports++;
    > +            struct lport_addresses lrp_addrs;
    > +            if (!extract_lrp_networks(lr->ports[i], &lrp_addrs)) {
    > +                static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
    > +                VLOG_WARN_RL(&rl, "Extract addresses failed.");
    > +                continue;
    > +            }
    > +
    > +            if (is_v6) {
    > +                for (int j = 0; j < lrp_addrs.n_ipv6_addrs; j++) {
    > +                    struct ipv6_netaddr *lrp6_addr = &lrp_addrs.ipv6_addrs[j];
    > +                    struct in6_addr new_ip6_mask, old_ip6_mask;
    > +                    new_ip6_mask = ipv6_addr_bitand(&lrp6_addr->mask,
    > +                                   &new_external_ip_addr.ipv6_addrs[0].addr);
    > +                    old_ip6_mask = ipv6_addr_bitand(&lrp6_addr->mask,
    > +                                   &old_external_ip_addr.ipv6_addrs[0].addr);
    > +                    if (ipv6_addr_equals(&new_ip6_mask, &(lrp6_addr->network))) {
    > +                        new_port = lrp;
    > +                    }
    > +                    if (ipv6_addr_equals(&old_ip6_mask, &(lrp6_addr->network))) {
    > +                        old_port = lrp;
    > +                    }
    > +                }
    > +            } else {
    > +                for (int j = 0; j < lrp_addrs.n_ipv4_addrs; j++) {
    > +                    uint32_t nw_addr = ntohl(lrp_addrs.ipv4_addrs[j].network);
    > +                    uint32_t mask = ntohl(lrp_addrs.ipv4_addrs[j].mask);
    > +                    uint32_t bcast = nw_addr | ~mask;
    > +                    uint32_t new_ip, old_ip;
    > +                    new_ip = ntohl(new_external_ip_addr.ipv4_addrs[0].addr);
    > +                    old_ip = ntohl(old_external_ip_addr.ipv4_addrs[0].addr);
    > +                    if (new_ip >= nw_addr && new_ip < bcast) {
    > +                        new_port = lrp;
    > +                    }
    > +                    if (old_ip >= nw_addr && old_ip < bcast) {
    > +                        old_port = lrp;
    > +                    }
    > +                }
    > +            }
    > +            if ((old_port || new_port) && (old_port == new_port)) {
    > +                is_conflict = true;
    > +            }
    > +            destroy_lport_addresses(&lrp_addrs);
    > +        }
    > +    }
    > +    destroy_lport_addresses(&old_external_ip_addr);
    > +    destroy_lport_addresses(&new_external_ip_addr);
    > +
    > +    if (num_l3dgw_ports > 1 && !is_conflict) {
    > +        return false;
    > +    }
    > +
    > +    return true;
    > +}
    > +
    >  static void
    >  nbctl_lr_nat_add(struct ctl_context *ctx)
    >  {
    > @@ -4446,12 +4530,16 @@ nbctl_lr_nat_add(struct ctl_context *ctx)
    >                              should_return = true;
    >                          }
    >                  } else {
    > -                    ctl_error(ctx, "a NAT with this type (%s) and %s (%s) "
    > -                              "already exists",
    > -                              nat_type,
    > -                              is_snat ? "logical_ip" : "external_ip",
    > -                              is_snat ? new_logical_ip : new_external_ip);
    > -                    should_return = true;
    > +                    if (!is_snat ||
    > +                        is_nat_rule_conflict(lr, new_external_ip,
    > +                                             old_external_ip, is_v6)) {
    > +                        ctl_error(ctx, "a NAT with this type (%s) and %s (%s) "
    > +                                  "already exists",
    > +                                  nat_type,
    > +                                  is_snat ? "logical_ip" : "external_ip",
    > +                                  is_snat ? new_logical_ip : new_external_ip);
    > +                        should_return = true;
    > +                    }
    >                  }
    >              }
    >          }
    > @@ -4547,6 +4635,19 @@ nbctl_lr_nat_del(struct ctl_context *ctx)
    >      }
    >
    >      int is_snat = !strcmp("snat", nat_type);
    > +    char *nat_external_ip = NULL;
    > +    if (ctx->argc == 5) {
    > +        if (is_snat) {
    > +            nat_external_ip = normalize_prefix_str(ctx->argv[4]);
    > +            if (!nat_external_ip) {
    > +                ctl_error(ctx, "%s: Invalid IP address or CIDR", ctx->argv[4]);
    > +            }
    > +        } else {
    > +            ctl_error(ctx, "%s type takes a maximum of one ip address", nat_type);
    > +        }
    > +    }
    > +    bool is_exist = false;
    > +
    >      /* Remove the matching NAT. */
    >      for (size_t i = 0; i < lr->n_nat; i++) {
    >          struct nbrec_nat *nat = lr->nat[i];
    > @@ -4558,8 +4659,29 @@ nbctl_lr_nat_del(struct ctl_context *ctx)
    >              continue;
    >          }
    >          if (!strcmp(nat_type, nat->type) && !strcmp(nat_ip, old_ip)) {
    > -            nbrec_logical_router_update_nat_delvalue(lr, nat);
    > -            should_return = true;
    > +            if (nat_external_ip != NULL) {
    > +                char *old_external_ip = normalize_prefix_str(nat->external_ip);
    > +                if (!old_external_ip) {
    > +                    continue;
    > +                }
    > +                if (!strcmp(nat_external_ip, old_external_ip)) {
    > +                    nbrec_logical_router_update_nat_delvalue(lr, nat);
    > +                    free(old_external_ip);
    > +                    is_exist = true;
    > +                    should_return = true;
    > +                }
    > +            } else {
    > +                nbrec_logical_router_update_nat_delvalue(lr, nat);
    > +                /* When nat_type is snat and external_ip is not specified, we
    > +                 * need to iterate over all the rules and delete all nat entries
    > +                 * matching the logical ip. Hence don't set should_return for
    > +                 * snat case.
    > +                 */
    > +                if (!is_snat) {
    > +                    should_return = true;
    > +                }
    > +                is_exist = true;
    > +            }
    >          }
    >          free(old_ip);
    >          if (should_return) {
    > @@ -4567,13 +4689,16 @@ nbctl_lr_nat_del(struct ctl_context *ctx)
    >          }
    >      }
    >
    > -    if (must_exist) {
    > +    if (must_exist && !is_exist) {
    >          ctl_error(ctx, "no matching NAT with the type (%s) and %s (%s)",
    >                    nat_type, is_snat ? "logical_ip" : "external_ip", nat_ip);
    >      }
    >
    >  cleanup:
    >      free(nat_ip);
    > +    if (nat_external_ip != NULL) {
    > +       free(nat_external_ip);
    > +    }
    >  }
    >
    >  static void
    > @@ -6635,7 +6760,7 @@ static const struct ctl_command_syntax nbctl_commands[] = {
    >        "ROUTER TYPE EXTERNAL_IP LOGICAL_IP"
    >        "[LOGICAL_PORT EXTERNAL_MAC] [EXTERNAL_PORT_RANGE]", NULL,
    >        nbctl_lr_nat_add, NULL, "--may-exist,--stateless,--portrange", RW },
    > -    { "lr-nat-del", 1, 3, "ROUTER [TYPE [IP]]", NULL,
    > +    { "lr-nat-del", 1, 4, "ROUTER [TYPE [IP [EXTERNAL_IP]]]", NULL,
    >          nbctl_lr_nat_del, NULL, "--if-exists", RW },
    >      { "lr-nat-list", 1, 1, "ROUTER", NULL, nbctl_lr_nat_list, NULL, "", RO },
    >      { "lr-nat-update-ext-ip", 4, 4, "ROUTER TYPE IP ADDRESS_SET", NULL,
    > --
    > 2.22.3
    >
    > _______________________________________________
    > dev mailing list
    > dev at openvswitch.org
    > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.openvswitch.org_mailman_listinfo_ovs-2Ddev&d=DwIFaQ&c=s883GpUCOChKOHiocYtGcg&r=dftLCwr11LPmmIAnjdakaot5N4f-Q2Qfwcs7XdgQZXE&m=qsTaQmkBjQHuap5egxV8-PeoiNS8HU-jA0qbn-WCVpQ&s=e2OyMaG826JbVPRwz5lwIe_bC5bbvNeyZO5CKdNTb3U&e= 
    >



More information about the dev mailing list