[ovs-dev] [PATCH v2 ovn] OVN: Multiple distributed gateway port support
Numan Siddique
numans at ovn.org
Wed Jan 27 18:14:07 UTC 2021
On Sat, Jan 16, 2021 at 3:53 AM Ankur Sharma <svc.mail.git at nutanix.com> wrote:
>
> From: Ankur Sharma <ankur.sharma at nutanix.com>
>
> By default, OVN support only one distributed gateway
> port (we will call it l3dgw port for further reference)
> per logical router. While a single l3dgw port suffices
> for most of the North South connectivity, however there
> are requirements where a logical router could be connected
> to multiple physical networks and based on routing decision
> packet could go to vlan X or vlan Y. Additionally, packet
> may or may not get NATed based on the configuration.
>
> This patch adds flexibility of having multiple l3dgw ports
> per logical router.
>
> Changes can classified as following:
> a. Data structure changes to allow multiple l3dgw ports per
> ovn_datapath.
>
> b. Consumption of new data structure in logical flows for
> individual features.
>
> c. Features that require changes are:
> i. Regular NS traffic flow.
> ii. Network Address Translation.
> iii. Load Balancer
> iv. Gateway_mtu.
> v. reside-on-redirect-chassis
> vi. Misc code sections that assumed a single l3dgw port.
>
> d. ovn-nbctl cli change to allow multiple external ips
> for a logical ip for same type.
>
> e. Except for reside-on-redirect-chassis all the other features
> could be extended to multiple l3dgw ports. Reside on redirect
> chassis with its current specification could not be extended
> and hence should be used only with the logical router that
> has a single l3dgw port.
>
> FUTURE WORK:
> CT ZONES are still common for traffic from different physical networks.
> This adds a restriction/assumption that same 5 tuple will not come
> from different l3dgw ports.
> A cleaner approach would be have different ct zones for each l3dgw port.
> Changing the CT ZONE assignment is one of the enhancements we are
> considering as next step.
>
> Signed-off-by: Ankur Sharma <ankur.sharma at nutanix.com>
> Signed-off-by: Dhathri Purohith <dhathri.purohith at nutanix.com>
> Signed-off-by: Ankur Sharma <ankurmnnit2004 at gmail.com>
> Co-authored-by: Dhathri Purohith <dhathri.purohith at nutanix.com>
> Co-authored-by: Ankur Sharma <ankurmnnit2004 at gmail.com>
Hi Ankur,
Thanks for adding this feature.
Since you're submitting this patch, I think you will be the primary
author. Probably
you can remove Co-authored-by tag with your name.
This patch needs a rebase. Can you please rebase it. I was the able to
resolve the conflicts
and apply the patch on my local repo on top of master. I see below
compilation issues with
sparse configured (./configure --enable-Werror --enable-sparse)
********
Tpo -c -o northd/ovn-northd.o ../northd/ovn-northd.c &&\
mv -f $depbase.Tpo $depbase.Po
../northd/ovn-northd.c:1406:13: error: incorrect type in assignment
(different base types)
../northd/ovn-northd.c:1406:13: expected restricted ovs_be32
[addressable] [usertype] ip4
../northd/ovn-northd.c:1406:13: got unsigned int
../northd/ovn-northd.c:1441:38: error: incorrect type in initializer
(different base types)
../northd/ovn-northd.c:1441:38: expected restricted ovs_be32 [usertype] addr
../northd/ovn-northd.c:1441:38: got unsigned int
../northd/ovn-northd.c:1442:41: error: incorrect type in initializer
(different base types)
../northd/ovn-northd.c:1442:41: expected restricted ovs_be32
[usertype] network
../northd/ovn-northd.c:1442:41: got unsigned int
../northd/ovn-northd.c:1443:23: error: incorrect type in assignment
(different base types)
../northd/ovn-northd.c:1443:23: expected restricted ovs_be32
[addressable] [usertype] mask4
../northd/ovn-northd.c:1443:23: got unsigned int
../northd/ovn-northd.c:1446:21: error: restricted ovs_be32 degrades to integer
../northd/ovn-northd.c:1446:28: error: restricted ovs_be32 degrades to integer
../northd/ovn-northd.c:1446:39: error: restricted ovs_be32 degrades to integer
../northd/ovn-northd.c:1446:45: error: restricted ovs_be32 degrades to integer
make[1]: *** [Makefile:2044: northd/ovn-northd.o] Error 1
*********
This is not a full review, but please see below with some comments.
Thanks
Numan
> ---
> northd/ovn-northd.c | 517 +++++++++++++++++++++++++++++++++-----------------
> tests/ovn-northd.at | 353 +++++++++++++++++++++++++++++++++-
> tests/ovn.at | 310 +++++++++++++++++++++++++++++-
> utilities/ovn-nbctl.c | 35 +++-
> 4 files changed, 1024 insertions(+), 191 deletions(-)
>
> diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c
> index 969fbe1..c1cef7d 100644
> --- a/northd/ovn-northd.c
> +++ b/northd/ovn-northd.c
> @@ -573,6 +573,19 @@ ovn_mcast_group_allocate_key(struct mcast_info *mcast_info)
> &mcast_info->group_tnlid_hint);
> }
>
> +struct ovn_datapath_l3dgw_port {
I'd suggest renaming this to 'struct ovn_l3dgw_port'.
> +
> + /* OVN northd only needs to know about the logical router gateway port for
> + * NAT on a distributed router. This "distributed gateway port" is
> + * populated only when there is a gateway chassis specified for one of
s/there is a gateway chassis/here is gateway chassis or ha chassis group
> + * the ports on the logical router. Otherwise this will be NULL. */
> + struct ovn_port *dgw_port;
> +
> + /* The "derived" OVN port representing the instance of l3dgw_port on
> + * the gateway chassis. */
> + struct ovn_port *redirect_port;
> +};
> +
> /* The 'key' comes from nbs->header_.uuid or nbr->header_.uuid or
> * sb->external_ids:logical-switch. */
> struct ovn_datapath {
> @@ -602,14 +615,9 @@ struct ovn_datapath {
> /* Multicast data. */
> struct mcast_info mcast_info;
>
> - /* OVN northd only needs to know about the logical router gateway port for
> - * NAT on a distributed router. This "distributed gateway port" is
> - * populated only when there is a gateway chassis specified for one of
> - * the ports on the logical router. Otherwise this will be NULL. */
> - struct ovn_port *l3dgw_port;
> - /* The "derived" OVN port representing the instance of l3dgw_port on
> - * the gateway chassis. */
> - struct ovn_port *l3redirect_port;
> + /* L3 distributed gateway ports */
> + struct ovn_datapath_l3dgw_port *l3dgw_ports;
> + size_t n_l3dgw_ports;
>
> /* NAT entries configured on the router. */
> struct ovn_nat *nat_entries;
> @@ -814,6 +822,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct ovn_datapath *od)
> ovn_destroy_tnlids(&od->port_tnlids);
> destroy_ipam_info(&od->ipam_info);
> free(od->router_ports);
> + free(od->l3dgw_ports);
> destroy_nat_entries(od);
> free(od->nat_entries);
> free(od->localnet_ports);
> @@ -1353,6 +1362,95 @@ struct ovn_port {
> struct ovs_list list; /* In list of similar records. */
> };
>
> +/* Get the l3dgw port corresponding to a logical router port.*/
> +static inline struct ovn_datapath_l3dgw_port*
> +ovn_get_l3dgw_port_from_lrp(const struct ovn_port *op)
> +{
> + struct ovn_datapath *od = op->od;
> +
> + if (!op || !op->nbrp) {
> + return NULL;
> + }
> +
> + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) {
Since n_l3dgw_ports is of type size_t, please use iter of the same type.
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + &(od->l3dgw_ports[iter]);
> + if (op == l3dgw_port->dgw_port) {
> + return l3dgw_port;
> + }
> + }
> +
> + return NULL;
> +}
> +
> +/* Get the l3dgw port corresponding to a logical router port
> + * with input ip */
> +static struct ovn_datapath_l3dgw_port*
> +ovn_get_l3dgw_port_from_ip(struct ovn_datapath *od, char *ip_s, bool is_v6)
(> +{
The caller of this function - build_lrouter_nat_defrag_and_lb(), has
already parsed the nat->external_ip
using ip_parse_masked or ipv6_parse_masked.
So I think there is no need to parse the same here. You can take
"struct in6_addr" as an argument. You
can use 'IN6_IS_ADDR_V4MAPPED' to determine if the 'struct in6_addr'
has IPv4 or IPv6 address.
build_lrouter_nat_defrag_and_lb() before calling this function -
ovn_get_l3dgw_port_from_ip(),
can use in6_addr_set_mapped_ipv4() to set IPv4 address.
> + ovs_be32 ip4, mask4;
> + struct in6_addr ip6, mask6;
> +
> + char *error = NULL;
> +
> + if (!od || !od->nbr) {
> + return NULL;
> + }
> +
> + if (is_v6) {
> + error = ipv6_parse_masked(ip_s, &ip6, &mask6);
> + } else {
> + error = ip_parse_masked(ip_s, &ip4, &mask4);
> + ip4 = ntohl(ip4);
> + }
> +
> + if (error) {
> + free(error);
> + return NULL;
> + }
> +
> + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + &(od->l3dgw_ports[iter]);
> + struct ovn_port *op = l3dgw_port->dgw_port;
> + struct lport_addresses lrp_networks;
> +
> + if (!extract_lrp_networks(op->nbrp, &lrp_networks)) {
> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> + VLOG_WARN_RL(&rl, "Extract addresses failed.");
> + continue;
> + }
> +
> + if (is_v6) {
> + for (int iter2 = 0; iter2 < lrp_networks.n_ipv6_addrs; iter2++) {
> + struct ipv6_netaddr *lrp6_addr =
> + &(lrp_networks.ipv6_addrs[iter2]);
> + struct in6_addr ip6_mask = ipv6_addr_bitand(&lrp6_addr->mask,
> + &ip6);
> +
> + if (ipv6_addr_equals(&ip6_mask, &(lrp6_addr->network))) {
> + return l3dgw_port;
> + }
> + }
> + } else {
> + for (int iter2 = 0; iter2 < lrp_networks.n_ipv4_addrs; iter2++) {
> + struct ipv4_netaddr *lrp4_addr =
> + &(lrp_networks.ipv4_addrs[iter2]);
> + ovs_be32 addr = ntohl(lrp4_addr->addr);
> + ovs_be32 network = ntohl(lrp4_addr->network);
> + mask4 = ntohl(lrp4_addr->mask);
> + ovs_be32 bcast = addr | ~mask4;
> +
> + if (ip4 >= network && ip4 < bcast) {
> + return l3dgw_port;
> + }
> + }
> + }
> + }
> +
> + return NULL;
> +}
> +
> static void
> ovn_port_set_nb(struct ovn_port *op,
> const struct nbrec_logical_switch_port *nbsp,
> @@ -2246,13 +2344,12 @@ join_logical_ports(struct northd_context *ctx,
> "on L3 gateway router", nbrp->name);
> continue;
> }
> - if (od->l3dgw_port || od->l3redirect_port) {
> + if (od->n_l3dgw_ports) {
> static struct vlog_rate_limit rl
> = VLOG_RATE_LIMIT_INIT(1, 1);
> - VLOG_WARN_RL(&rl, "Bad configuration: multiple "
> - "distributed gateway ports on logical "
> - "router %s", od->nbr->name);
> - continue;
> + VLOG_DBG_RL(&rl, "Multiple ports with "
> + "redirect-chassis on same "
> + "logical router %s", od->nbr->name);
> }
>
> char *redirect_name =
> @@ -2274,8 +2371,12 @@ join_logical_ports(struct northd_context *ctx,
>
> /* Set l3dgw_port and l3redirect_port in od, for later
> * use during flow creation. */
> - od->l3dgw_port = op;
> - od->l3redirect_port = crp;
> + od->l3dgw_ports = xrealloc(od->l3dgw_ports,
> + sizeof *od->l3dgw_ports *
> + (od->n_l3dgw_ports + 1));
> + (od->l3dgw_ports[od->n_l3dgw_ports]).dgw_port = op;
> + (od->l3dgw_ports[od->n_l3dgw_ports]).redirect_port = crp;
> + od->n_l3dgw_ports++;
> }
> }
> }
> @@ -2431,7 +2532,7 @@ get_nat_addresses(const struct ovn_port *op, size_t *n)
>
> /* Determine whether this NAT rule satisfies the conditions for
> * distributed NAT processing. */
> - if (op->od->l3redirect_port && !strcmp(nat->type, "dnat_and_snat")
> + if (op->od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat")
> && nat->logical_port && nat->external_mac) {
> /* Distributed NAT rule. */
> if (eth_addr_from_string(nat->external_mac, &mac)) {
> @@ -2491,11 +2592,13 @@ get_nat_addresses(const struct ovn_port *op, size_t *n)
> sset_destroy(&all_ips_v6);
>
> if (central_ip_address) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> /* Gratuitous ARP for centralized NAT rules on distributed gateway
> * ports should be restricted to the gateway chassis. */
> - if (op->od->l3redirect_port) {
> + if (l3dgw_port) {
> ds_put_format(&c_addresses, " is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
>
> addresses[n_nats++] = ds_steal_cstr(&c_addresses);
> @@ -2988,7 +3091,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> char **nats = NULL;
> if (nat_addresses && !strcmp(nat_addresses, "router")) {
> if (op->peer && op->peer->od
> - && (chassis || op->peer->od->l3redirect_port)) {
> + && (chassis || op->peer->od->n_l3dgw_ports)) {
> nats = get_nat_addresses(op->peer, &n_nats);
> }
> /* Only accept manual specification of ethernet address
> @@ -3024,11 +3127,11 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> * sending the GARPs for the router port IPs.
> * */
> bool add_router_port_garp = false;
> - if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_port &&
> - op->peer->od->l3redirect_port &&
> + struct ovn_datapath_l3dgw_port *l3dgw_port = NULL;
> + if (op->peer && op->peer->nbrp && op->peer->od->n_l3dgw_ports &&
> (smap_get_bool(&op->peer->nbrp->options,
> "reside-on-redirect-chassis", false) ||
> - op->peer == op->peer->od->l3dgw_port)) {
> + (l3dgw_port = ovn_get_l3dgw_port_from_lrp(op->peer)))) {
> add_router_port_garp = true;
> } else if (chassis && op->od->n_localnet_ports) {
> add_router_port_garp = true;
> @@ -3043,9 +3146,12 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> op->peer->lrp_networks.ipv4_addrs[i].addr_s);
> }
>
> - if (op->peer->od->l3redirect_port) {
> + if (op->peer->od->n_l3dgw_ports) {
> + if (!l3dgw_port) {
> + l3dgw_port = &(op->peer->od->l3dgw_ports[0]);
> + }
> ds_put_format(&garp_info, " is_chassis_resident(%s)",
> - op->peer->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
>
> n_nats++;
> @@ -4637,7 +4743,6 @@ build_lswitch_input_port_sec_op(
> struct ovn_port *op, struct hmap *lflows,
> struct ds *actions, struct ds *match)
> {
> -
> if (!op->nbsp) {
> return;
> }
> @@ -5937,13 +6042,16 @@ build_lrouter_groups__(struct hmap *ports, struct ovn_datapath *od)
> {
> ovs_assert((od && od->nbr && od->lr_group));
>
> - if (od->l3dgw_port && od->l3redirect_port) {
> + for (int i = 0; i < od->n_l3dgw_ports; i++) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + &(od->l3dgw_ports[i]);
> +
> /* It's a logical router with gateway port. If it
> * has HA_Chassis_Group associated to it in SB DB, then store the
> * ha chassis group name. */
> - if (od->l3redirect_port->sb->ha_chassis_group) {
> + if (l3dgw_port->redirect_port->sb->ha_chassis_group) {
> sset_add(&od->lr_group->ha_chassis_groups,
> - od->l3redirect_port->sb->ha_chassis_group->name);
> + l3dgw_port->redirect_port->sb->ha_chassis_group->name);
> }
> }
>
> @@ -7182,17 +7290,18 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
> ds_clear(match);
> ds_put_format(match, "eth.dst == "ETH_ADDR_FMT,
> ETH_ADDR_ARGS(mac));
> - if (op->peer->od->l3dgw_port
> - && op->peer->od->l3redirect_port
> - && op->od->n_localnet_ports) {
> + if (op->peer->od->n_l3dgw_ports &&
> + op->od->n_localnet_ports) {
> bool add_chassis_resident_check = false;
> - if (op->peer == op->peer->od->l3dgw_port) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op->peer);
> + if (l3dgw_port) {
> /* The peer of this port represents a distributed
> * gateway port. The destination lookup flow for the
> * router's distributed gateway port MAC address should
> * only be programmed on the gateway chassis. */
> add_chassis_resident_check = true;
> - } else {
> + } else if (op->peer->od->n_l3dgw_ports == 1) {
> /* Check if the option 'reside-on-redirect-chassis'
> * is set to true on the peer port. If set to true
> * and if the logical switch has a localnet port, it
> @@ -7200,6 +7309,11 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
> * this logical switch should be run on the chassis
> * hosting the gateway port.
> */
> +
> + /* reside-on-redirect-chassis is supported only for
> + * logical routers with single l3dgw port.
> + */
> + l3dgw_port = &(op->peer->od->l3dgw_ports[0]);
> add_chassis_resident_check = smap_get_bool(
> &op->peer->nbrp->options,
> "reside-on-redirect-chassis", false);
> @@ -7207,7 +7321,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
> if (add_chassis_resident_check) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->peer->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> }
>
> @@ -7220,8 +7334,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
> /* Add ethernet addresses specified in NAT rules on
> * distributed logical routers. */
> - if (op->peer->od->l3dgw_port
> - && op->peer == op->peer->od->l3dgw_port) {
> + if (ovn_get_l3dgw_port_from_lrp(op->peer)) {
> for (int j = 0; j < op->peer->od->nbr->n_nat; j++) {
> const struct nbrec_nat *nat
> = op->peer->od->nbr->nat[j];
> @@ -8357,34 +8470,44 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od,
> build_empty_lb_event_flow(od, lflows, lb_vip, lb, S_ROUTER_IN_DNAT,
> meter_groups);
>
> - /* A match and actions for new connections. */
> - char *new_match = xasprintf("ct.new && %s", ds_cstr(match));
> - if (lb_force_snat_ip) {
> - char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s",
> - ds_cstr(actions));
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> - new_match, new_actions, &lb->header_);
> - free(new_actions);
> - } else {
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> - new_match, ds_cstr(actions), &lb->header_);
> - }
> + for (int i = 0; i < od->n_l3dgw_ports; i++) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port = &(od->l3dgw_ports[i]);
> +
> + /* A match and actions for new connections. */
> + char *new_match = xasprintf("ct.new && %s && inport == %s && "
> + "is_chassis_resident(%s)", ds_cstr(match),
> + l3dgw_port->dgw_port->json_key,
> + l3dgw_port->redirect_port->json_key);
> + if (lb_force_snat_ip) {
> + char *new_actions = xasprintf("flags.force_snat_for_lb = 1; %s",
> + ds_cstr(actions));
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> + new_match, new_actions, &lb->header_);
> + free(new_actions);
> + } else {
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> + new_match, ds_cstr(actions), &lb->header_);
> + }
> +
> + /* A match and actions for established connections. */
> + char *est_match = xasprintf("ct.est && %s && inport == %s && "
> + "is_chassis_resident(%s)", ds_cstr(match),
> + l3dgw_port->dgw_port->json_key,
> + l3dgw_port->redirect_port->json_key);
> + if (lb_force_snat_ip) {
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> + est_match,
> + "flags.force_snat_for_lb = 1; ct_dnat;",
> + &lb->header_);
> + } else {
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> + est_match, "ct_dnat;", &lb->header_);
> + }
>
> - /* A match and actions for established connections. */
> - char *est_match = xasprintf("ct.est && %s", ds_cstr(match));
> - if (lb_force_snat_ip) {
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> - est_match,
> - "flags.force_snat_for_lb = 1; ct_dnat;",
> - &lb->header_);
> - } else {
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_DNAT, priority,
> - est_match, "ct_dnat;", &lb->header_);
> + free(new_match);
> + free(est_match);
> }
>
> - free(new_match);
> - free(est_match);
> -
> const char *ip_match = NULL;
> if (IN6_IS_ADDR_V4MAPPED(&lb_vip->vip)) {
> ip_match = "ip4";
> @@ -8418,49 +8541,55 @@ add_router_lb_flow(struct hmap *lflows, struct ovn_datapath *od,
> ds_destroy(&unsnat_match);
> }
>
> - if (!od->l3dgw_port || !od->l3redirect_port || !lb_vip->n_backends) {
> + if (!od->n_l3dgw_ports || !lb_vip->n_backends) {
> return;
> }
>
> - /* Add logical flows to UNDNAT the load balanced reverse traffic in
> - * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the logical
> - * router has a gateway router port associated.
> - */
> - struct ds undnat_match = DS_EMPTY_INITIALIZER;
> - ds_put_format(&undnat_match, "%s && (", ip_match);
> + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) {
> +
> + /* Add logical flows to UNDNAT the load balanced reverse traffic in
> + * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the
> + * logical router has a gateway router port associated.
> + */
> + struct ds undnat_match = DS_EMPTY_INITIALIZER;
> + ds_put_format(&undnat_match, "%s && (", ip_match);
> +
> + struct ovn_datapath_l3dgw_port *l3dgw_port = &(od->l3dgw_ports[iter]);
>
> - for (size_t i = 0; i < lb_vip->n_backends; i++) {
> - struct ovn_lb_backend *backend = &lb_vip->backends[i];
> - ds_put_format(&undnat_match, "(%s.src == %s", ip_match,
> - backend->ip_str);
> + for (size_t i = 0; i < lb_vip->n_backends; i++) {
> + struct ovn_lb_backend *backend = &lb_vip->backends[i];
> + ds_put_format(&undnat_match, "(%s.src == %s", ip_match,
> + backend->ip_str);
>
> - if (backend->port) {
> - ds_put_format(&undnat_match, " && %s.src == %d) || ",
> - proto, backend->port);
> + if (backend->port) {
> + ds_put_format(&undnat_match, " && %s.src == %d) || ",
> + proto, backend->port);
> + } else {
> + ds_put_cstr(&undnat_match, ") || ");
> + }
> + }
> +
> + ds_chomp(&undnat_match, ' ');
> + ds_chomp(&undnat_match, '|');
> + ds_chomp(&undnat_match, '|');
> + ds_chomp(&undnat_match, ' ');
> + ds_put_format(&undnat_match, ") && outport == %s && "
> + "is_chassis_resident(%s)",
> + l3dgw_port->dgw_port->json_key,
> + l3dgw_port->redirect_port->json_key);
> + if (lb_force_snat_ip) {
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> + ds_cstr(&undnat_match),
> + "flags.force_snat_for_lb = 1; ct_dnat;",
> + &lb->header_);
> } else {
> - ds_put_cstr(&undnat_match, ") || ");
> - }
> - }
> -
> - ds_chomp(&undnat_match, ' ');
> - ds_chomp(&undnat_match, '|');
> - ds_chomp(&undnat_match, '|');
> - ds_chomp(&undnat_match, ' ');
> - ds_put_format(&undnat_match, ") && outport == %s && "
> - "is_chassis_resident(%s)", od->l3dgw_port->json_key,
> - od->l3redirect_port->json_key);
> - if (lb_force_snat_ip) {
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> - ds_cstr(&undnat_match),
> - "flags.force_snat_for_lb = 1; ct_dnat;",
> - &lb->header_);
> - } else {
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> - ds_cstr(&undnat_match), "ct_dnat;",
> - &lb->header_);
> - }
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> + ds_cstr(&undnat_match), "ct_dnat;",
> + &lb->header_);
> + }
>
> - ds_destroy(&undnat_match);
> + ds_destroy(&undnat_match);
> + }
> }
>
> #define ND_RA_MAX_INTERVAL_MAX 1800
> @@ -8582,7 +8711,7 @@ lrouter_nat_add_ext_ip_match(struct ovn_datapath *od,
> {
> struct nbrec_address_set *allowed_ext_ips = nat->allowed_ext_ips;
> struct nbrec_address_set *exempted_ext_ips = nat->exempted_ext_ips;
> - bool is_gw_router = !od->l3dgw_port;
> + bool is_gw_router = !od->n_l3dgw_ports;
>
> ovs_assert(allowed_ext_ips || exempted_ext_ips);
>
> @@ -8797,9 +8926,11 @@ build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op,
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ARP responses
> * from different chassis. */
> - if (op->od->l3redirect_port) {
> + if (op->od->n_l3dgw_ports) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> ds_put_format(&match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> }
>
> @@ -8986,6 +9117,8 @@ build_adm_ctrl_flows_for_lrouter_port(
> struct ovn_port *op, struct hmap *lflows,
> struct ds *match, struct ds *actions)
> {
> + struct ovn_datapath_l3dgw_port *l3dgw_port = NULL;
> +
> if (op->nbrp) {
> if (!lrport_is_enabled(op->nbrp)) {
> /* Drop packets from disabled logical ports (since logical flow
> @@ -9016,12 +9149,12 @@ build_adm_ctrl_flows_for_lrouter_port(
> ds_clear(match);
> ds_put_format(match, "eth.dst == %s && inport == %s",
> op->lrp_networks.ea_s, op->json_key);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + l3dgw_port = ovn_get_l3dgw_port_from_lrp(op);
> + if (l3dgw_port) {
> /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s
> * should only be received on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> ovn_lflow_add_with_hint(lflows, op->od, S_ROUTER_IN_ADMISSION, 50,
> ds_cstr(match), ds_cstr(actions),
> @@ -9141,6 +9274,8 @@ build_neigh_learning_flows_for_lrouter_port(
>
> /* Check if we need to learn mac-binding from ARP requests. */
> for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> if (!learn_from_arp_request) {
> /* ARP request to this address should always get learned,
> * so add a priority-110 flow to set
> @@ -9153,10 +9288,9 @@ build_neigh_learning_flows_for_lrouter_port(
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen,
> op->lrp_networks.ipv4_addrs[i].addr_s);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (l3dgw_port) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> const char *actions_s = REGBIT_LOOKUP_NEIGHBOR_RESULT
> " = lookup_arp(inport, arp.spa, arp.sha); "
> @@ -9173,10 +9307,9 @@ build_neigh_learning_flows_for_lrouter_port(
> op->json_key,
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (l3dgw_port) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> ds_clear(actions);
> ds_put_format(actions, REGBIT_LOOKUP_NEIGHBOR_RESULT
> @@ -9601,7 +9734,10 @@ build_arp_resolve_flows_for_lrouter_port(
> }
> }
>
> - if (!op->derived && op->od->l3redirect_port) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> +
> + if (!op->derived && l3dgw_port) {
> const char *redirect_type = smap_get(&op->nbrp->options,
> "redirect-type");
> if (redirect_type && !strcasecmp(redirect_type, "bridged")) {
> @@ -9614,7 +9750,7 @@ build_arp_resolve_flows_for_lrouter_port(
> ds_clear(match);
> ds_put_format(match, "outport == %s && "
> "!is_chassis_resident(%s)", op->json_key,
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> ds_clear(actions);
> ds_put_format(actions, "eth.dst = %s; next;",
> op->lrp_networks.ea_s);
> @@ -9925,11 +10061,14 @@ build_check_pkt_len_flows_for_lrouter(
> ovn_lflow_add(lflows, od, S_ROUTER_IN_LARGER_PKTS, 0, "1",
> "next;");
>
> - if (od->l3dgw_port && od->l3redirect_port) {
> + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) {
> int gw_mtu = 0;
> - if (od->l3dgw_port->nbrp) {
> - gw_mtu = smap_get_int(&od->l3dgw_port->nbrp->options,
> - "gateway_mtu", 0);
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + &(od->l3dgw_ports[iter]);
> +
> + if (l3dgw_port->dgw_port->nbrp) {
> + gw_mtu = smap_get_int(&(l3dgw_port->dgw_port->nbrp->options),
> + "gateway_mtu", 0);
> }
> /* Add the flows only if gateway_mtu is configured. */
> if (gw_mtu <= 0) {
> @@ -9937,20 +10076,20 @@ build_check_pkt_len_flows_for_lrouter(
> }
>
> ds_clear(match);
> - ds_put_format(match, "outport == %s", od->l3dgw_port->json_key);
> -
> + ds_put_format(match, "outport == %s",
> + l3dgw_port->dgw_port->json_key);
> ds_clear(actions);
> ds_put_format(actions,
> REGBIT_PKT_LARGER" = check_pkt_larger(%d);"
> " next;", gw_mtu + VLAN_ETH_HEADER_LEN);
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_CHK_PKT_LEN, 50,
> ds_cstr(match), ds_cstr(actions),
> - &od->l3dgw_port->nbrp->header_);
> + &(l3dgw_port->dgw_port->nbrp->header_));
>
> for (size_t i = 0; i < od->nbr->n_ports; i++) {
> struct ovn_port *rp = ovn_port_find(ports,
> od->nbr->ports[i]->name);
> - if (!rp || rp == od->l3dgw_port) {
> + if (rp == l3dgw_port->dgw_port) {
> continue;
> }
>
> @@ -9958,7 +10097,8 @@ build_check_pkt_len_flows_for_lrouter(
> ds_clear(match);
> ds_put_format(match, "inport == %s && outport == %s"
> " && ip4 && "REGBIT_PKT_LARGER,
> - rp->json_key, od->l3dgw_port->json_key);
> + rp->json_key,
> + l3dgw_port->dgw_port->json_key);
>
> ds_clear(actions);
> /* Set icmp4.frag_mtu to gw_mtu */
> @@ -9987,7 +10127,8 @@ build_check_pkt_len_flows_for_lrouter(
> ds_clear(match);
> ds_put_format(match, "inport == %s && outport == %s"
> " && ip6 && "REGBIT_PKT_LARGER,
> - rp->json_key, od->l3dgw_port->json_key);
> + rp->json_key, l3dgw_port->dgw_port
> + ->json_key);
>
> ds_clear(actions);
> /* Set icmp6.frag_mtu to gw_mtu */
> @@ -10029,11 +10170,14 @@ build_gateway_redirect_flows_for_lrouter(
> struct ds *match, struct ds *actions)
> {
> if (od->nbr) {
> - if (od->l3dgw_port && od->l3redirect_port) {
> + for (int iter = 0; iter < od->n_l3dgw_ports; iter++) {
> const struct ovsdb_idl_row *stage_hint = NULL;
>
> - if (od->l3dgw_port->nbrp) {
> - stage_hint = &od->l3dgw_port->nbrp->header_;
> + struct ovn_port *l3dgw_port = (od->l3dgw_ports[iter]).dgw_port;
> + struct ovn_port *l3redirect_port =
> + (od->l3dgw_ports[iter]).redirect_port;
> + if (l3dgw_port->nbrp) {
> + stage_hint = &l3dgw_port->nbrp->header_;
> }
>
> /* For traffic with outport == l3dgw_port, if the
> @@ -10041,13 +10185,12 @@ build_gateway_redirect_flows_for_lrouter(
> * rule, then the traffic is redirected to the central
> * instance of the l3dgw_port. */
> ds_clear(match);
> - ds_put_format(match, "outport == %s",
> - od->l3dgw_port->json_key);
> + ds_put_format(match, "outport == %s", l3dgw_port->json_key);
> ds_clear(actions);
> ds_put_format(actions, "outport = %s; next;",
> - od->l3redirect_port->json_key);
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50,
> - ds_cstr(match), ds_cstr(actions),
> + l3redirect_port->json_key);
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT,
> + 50, ds_cstr(match), ds_cstr(actions),
> stage_hint);
> }
>
> @@ -10278,16 +10421,17 @@ build_ipv6_input_flows_for_lrouter_port(
> /* ND reply. These flows reply to ND solicitations for the
> * router's own IP address. */
> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> ds_clear(match);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (l3dgw_port) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
> * should only be sent from the gateway chassi, so that
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ND replies
> * from different chassis. */
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
>
> build_lrouter_nd_flow(op->od, op, "nd_na_router",
> @@ -10299,7 +10443,7 @@ build_ipv6_input_flows_for_lrouter_port(
>
> /* UDP/TCP/SCTP port unreachable */
> if (!smap_get(&op->od->nbr->options, "chassis")
> - && !op->od->l3dgw_port) {
> + && !op->od->n_l3dgw_ports) {
> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> ds_clear(match);
> ds_put_format(match,
> @@ -10515,10 +10659,12 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen);
>
> - if (op->od->l3dgw_port && op->od->l3redirect_port && op->peer
> + if (op->od->n_l3dgw_ports && op->peer
> && op->peer->od->n_localnet_ports) {
> bool add_chassis_resident_check = false;
> - if (op == op->od->l3dgw_port) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> + if (l3dgw_port) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
> * should only be sent from the gateway chassis, so that
> * upstream MAC learning points to the gateway chassis.
> @@ -10534,6 +10680,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> * hosting the gateway port and it should reply to the
> * ARP requests for the router port IPs.
> */
> + l3dgw_port = &(op->od->l3dgw_ports[0]);
> add_chassis_resident_check = smap_get_bool(
> &op->nbrp->options,
> "reside-on-redirect-chassis", false);
> @@ -10541,7 +10688,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
>
> if (add_chassis_resident_check) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> }
>
> @@ -10559,9 +10706,11 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> const char *ip_address;
> SSET_FOR_EACH (ip_address, &all_ips_v4) {
> ds_clear(match);
> - if (op == op->od->l3dgw_port) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> + if (l3dgw_port) {
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
>
> build_lrouter_arp_flow(op->od, op,
> @@ -10571,9 +10720,11 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
>
> SSET_FOR_EACH (ip_address, &all_ips_v6) {
> ds_clear(match);
> - if (op == op->od->l3dgw_port) {
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + ovn_get_l3dgw_port_from_lrp(op);
> + if (l3dgw_port) {
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
>
> build_lrouter_nd_flow(op->od, op, "nd_na",
> @@ -10585,7 +10736,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> sset_destroy(&all_ips_v6);
>
> if (!smap_get(&op->od->nbr->options, "chassis")
> - && !op->od->l3dgw_port) {
> + && !op->od->n_l3dgw_ports) {
> /* UDP/TCP/SCTP port unreachable. */
> for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
> ds_clear(match);
> @@ -10662,7 +10813,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> * exception is on the l3dgw_port where we might need to use a
> * different ETH address.
> */
> - if (op != op->od->l3dgw_port) {
> + if (!ovn_get_l3dgw_port_from_lrp(op)) {
> return;
> }
>
> @@ -10727,7 +10878,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> /* NAT rules are only valid on Gateway routers and routers with
> * l3dgw_port (router has a port with gateway chassis
> * specified). */
> - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) {
> + if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) {
> return;
> }
>
> @@ -10814,11 +10965,27 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> }
> }
>
> + struct ovn_datapath_l3dgw_port *l3dgw_port =
> + &((od)->l3dgw_ports[0]);
> + if (od->n_l3dgw_ports) {
> + /* Get the L3DGW port only for distributed router. */
> + l3dgw_port = ovn_get_l3dgw_port_from_ip(od, nat->external_ip,
> + is_v6);
> + if (!l3dgw_port) {
> + static struct vlog_rate_limit rl =
> + VLOG_RATE_LIMIT_INIT(5, 1);
> + VLOG_WARN_RL(&rl, "Could not map external ip: %s to a "
> + "gateway port "UUID_FMT"", nat->external_ip,
> + UUID_ARGS(&od->key));
> + continue;
> + }
> + }
> +
> /* For distributed router NAT, determine whether this NAT rule
> * satisfies the conditions for distributed NAT processing. */
> bool distributed = false;
> struct eth_addr mac;
> - if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") &&
> + if (od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") &&
> nat->logical_port && nat->external_mac) {
> if (eth_addr_from_string(nat->external_mac, &mac)) {
> distributed = true;
> @@ -10842,7 +11009,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> * egress pipeline. */
> if (!strcmp(nat->type, "snat")
> || !strcmp(nat->type, "dnat_and_snat")) {
> - if (!od->l3dgw_port) {
> + if (!od->n_l3dgw_ports) {
> /* Gateway router. */
> ds_clear(match);
> ds_clear(actions);
> @@ -10870,12 +11037,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> " && inport == %s",
> is_v6 ? "6" : "4",
> nat->external_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + l3dgw_port->dgw_port->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
>
> if (!strcmp(nat->type, "dnat_and_snat") && stateless) {
> @@ -10897,7 +11064,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> * to a logical IP address. */
> if (!strcmp(nat->type, "dnat")
> || !strcmp(nat->type, "dnat_and_snat")) {
> - if (!od->l3dgw_port) {
> + if (!od->n_l3dgw_ports) {
> /* Gateway router. */
> /* Packet when it goes from the initiator to destination.
> * We need to set flags.loopback because the router can
> @@ -10947,12 +11114,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> " && inport == %s",
> is_v6 ? "6" : "4",
> nat->external_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + l3dgw_port->dgw_port->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> ds_clear(actions);
> if (allowed_ext_ips || exempted_ext_ips) {
> @@ -10979,12 +11146,12 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> }
>
> /* ARP resolve for NAT IPs. */
> - if (od->l3dgw_port) {
> + if (od->n_l3dgw_ports) {
> if (!strcmp(nat->type, "snat")) {
> ds_clear(match);
> ds_put_format(
> match, "inport == %s && %s == %s",
> - od->l3dgw_port->json_key,
> + l3dgw_port->dgw_port->json_key,
> is_v6 ? "ip6.src" : "ip4.src", nat->external_ip);
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_INPUT,
> 120, ds_cstr(match), "next;",
> @@ -10995,14 +11162,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> ds_clear(match);
> ds_put_format(
> match, "outport == %s && %s == %s",
> - od->l3dgw_port->json_key,
> + l3dgw_port->dgw_port->json_key,
> is_v6 ? REG_NEXT_HOP_IPV6 : REG_NEXT_HOP_IPV4,
> nat->external_ip);
> ds_clear(actions);
> ds_put_format(
> actions, "eth.dst = %s; next;",
> distributed ? nat->external_mac :
> - od->l3dgw_port->lrp_networks.ea_s);
> + l3dgw_port->dgw_port->lrp_networks.ea_s);
> ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_IN_ARP_RESOLVE,
> 100, ds_cstr(match),
> @@ -11025,19 +11192,19 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> * Note that this only applies for NAT on a distributed router.
> * Undo DNAT on a gateway router is done in the ingress DNAT
> * pipeline stage. */
> - if (od->l3dgw_port && (!strcmp(nat->type, "dnat")
> + if (od->n_l3dgw_ports && (!strcmp(nat->type, "dnat")
> || !strcmp(nat->type, "dnat_and_snat"))) {
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.src == %s"
> " && outport == %s",
> is_v6 ? "6" : "4",
> nat->logical_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + l3dgw_port->dgw_port->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> ds_clear(actions);
> if (distributed) {
> @@ -11062,7 +11229,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> * address. */
> if (!strcmp(nat->type, "snat")
> || !strcmp(nat->type, "dnat_and_snat")) {
> - if (!od->l3dgw_port) {
> + if (!od->n_l3dgw_ports) {
> /* Gateway router. */
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.src == %s",
> @@ -11105,13 +11272,13 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> " && outport == %s",
> is_v6 ? "6" : "4",
> nat->logical_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + l3dgw_port->dgw_port->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> priority += 128;
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> }
> ds_clear(actions);
>
> @@ -11160,14 +11327,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> */
> ds_clear(actions);
> ds_put_format(actions, REG_INPORT_ETH_ADDR " = %s; next;",
> - od->l3dgw_port->lrp_networks.ea_s);
> + l3dgw_port->dgw_port->lrp_networks.ea_s);
>
> ds_clear(match);
> ds_put_format(match,
> "eth.dst == "ETH_ADDR_FMT" && inport == %s"
> " && is_chassis_resident(\"%s\")",
> ETH_ADDR_ARGS(mac),
> - od->l3dgw_port->json_key,
> + l3dgw_port->dgw_port->json_key,
> nat->logical_port);
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ADMISSION, 50,
> ds_cstr(match), ds_cstr(actions),
> @@ -11190,7 +11357,8 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> "ip%s.src == %s && outport == %s && "
> "is_chassis_resident(\"%s\")",
> is_v6 ? "6" : "4", nat->logical_ip,
> - od->l3dgw_port->json_key, nat->logical_port);
> + l3dgw_port->dgw_port->json_key,
> + nat->logical_port);
> ds_put_format(actions, "eth.src = %s; %s = %s; next;",
> nat->external_mac,
> is_v6 ? REG_SRC_IPV6 : REG_SRC_IPV4,
> @@ -11205,16 +11373,16 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> * gateway port have ip.dst matching a NAT external IP, then
> * loop a clone of the packet back to the beginning of the
> * ingress pipeline with inport = outport. */
> - if (od->l3dgw_port) {
> + if (od->n_l3dgw_ports) {
> /* Distributed router. */
> ds_clear(match);
> ds_put_format(match, "ip%s.dst == %s && outport == %s",
> is_v6 ? "6" : "4",
> nat->external_ip,
> - od->l3dgw_port->json_key);
> + l3dgw_port->dgw_port->json_key);
> if (!distributed) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + l3dgw_port->redirect_port->json_key);
> } else {
> ds_put_format(match, " && is_chassis_resident(\"%s\")",
> nat->logical_port);
> @@ -11238,7 +11406,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> }
>
> /* Handle force SNAT options set in the gateway router. */
> - if (!od->l3dgw_port) {
> + if (!od->n_l3dgw_ports) {
> if (dnat_force_snat_ip) {
> if (od->dnat_force_snat_addrs.n_ipv4_addrs) {
> build_lrouter_force_snat_flows(lflows, od, "4",
> @@ -11277,7 +11445,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
>
> /* Load balancing and packet defrag are only valid on
> * Gateway routers or router with gateway port. */
> - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) {
> + if (!smap_get(&od->nbr->options, "chassis") && !od->n_l3dgw_ports) {
> sset_destroy(&nat_entries);
> return;
> }
> @@ -11347,11 +11515,6 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od,
> prio = 120;
> }
>
> - if (od->l3redirect_port &&
> - (lb_vip->n_backends || !lb_vip->empty_backend_rej)) {
> - ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> - }
> add_router_lb_flow(lflows, od, match, actions, prio,
> lb_force_snat_ip, lb_vip, proto,
> nb_lb, meter_groups, &nat_entries);
> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> index 91eb9a3..2c01f20 100644
> --- a/tests/ovn-northd.at
> +++ b/tests/ovn-northd.at
> @@ -711,7 +711,7 @@ ovn_start
> ovn-sbctl chassis-add gw1 geneve 127.0.0.1
>
> ovn-nbctl lr-add R1
> -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64
>
> ovn-nbctl ls-add S1
> ovn-nbctl lsp-add S1 S1-R1
> @@ -752,13 +752,13 @@ ovn-nbctl lr-nat-del R1 dnat_and_snat 172.16.1.1
>
> echo
> echo "IPv6: stateful"
> -ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> +ovn-nbctl --wait=sb lr-nat-add R1 dnat_and_snat 3000::c 1000::3
> check_flow_match_sets 2 2 2 0 0 0 0
> -ovn-nbctl lr-nat-del R1 dnat_and_snat fd01::1
> +ovn-nbctl lr-nat-del R1 dnat_and_snat 3000::c
>
> echo
> echo "IPv6: stateless"
> -ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat fd01::1 fd11::2
> +ovn-nbctl --wait=sb --stateless lr-nat-add R1 dnat_and_snat 3000::c 1000::3
> check_flow_match_sets 2 0 0 0 0 2 2
>
> AT_CLEANUP
> @@ -769,7 +769,7 @@ ovn_start
> ovn-sbctl chassis-add gw1 geneve 127.0.0.1
>
> ovn-nbctl lr-add R1
> -ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add R1 R1-S1 02:ac:10:01:00:01 172.16.1.1/24 3000::a/64
>
> ovn-nbctl ls-add S1
> ovn-nbctl lsp-add S1 S1-R1
> @@ -2385,3 +2385,346 @@ ovn-nbctl destroy bfd $uuid
> check_row_count bfd 2
>
> AT_CLEANUP
> +
> +AT_SETUP([ovn-northd -- lr multiple gw ports])
> +ovn_start
> +
> +# Logical network:
> +# 1 LR, 3 Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# We will validate basic LR logical flows.
> +
> +ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> +ovn-sbctl chassis-add gw2 geneve 128.0.0.1
> +ovn-sbctl chassis-add gw3 geneve 129.0.0.1
> +
> +ovn-nbctl lr-add DR
> +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.0/24
> +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.0/24
> +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24
> +
> +ovn-nbctl ls-add S1
> +ovn-nbctl lsp-add S1 S1-DR
> +ovn-nbctl lsp-set-type S1-DR router
> +ovn-nbctl lsp-set-addresses S1-DR router
> +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +
> +ovn-nbctl ls-add S2
> +ovn-nbctl lsp-add S2 S2-DR
> +ovn-nbctl lsp-set-type S2-DR router
> +ovn-nbctl lsp-set-addresses S2-DR router
> +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +
> +ovn-nbctl ls-add S3
> +ovn-nbctl lsp-add S3 S3-DR
> +ovn-nbctl lsp-set-type S3-DR router
> +ovn-nbctl lsp-set-addresses S3-DR router
> +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +
> +ovn-nbctl ls-add ls
> +ovn-nbctl lsp-add ls ls-DR
> +ovn-nbctl lsp-set-type ls-DR router
> +ovn-nbctl lsp-set-addresses ls-DR router
> +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
> +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
> +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
> +
> +ovn-nbctl --wait=sb sync
> +
> +ovn-sbctl dump-flows DR
> +
> +# Check the flows in lr_in_lookup_neighbor stage
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR | wc -l], [0], [3
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S1 | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S2 | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_lookup_neighbor | grep cr-DR-S3 | wc -l], [0], [1
> +])
> +
> +# Check the flows in lr_in_gw_redirect stage
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR | wc -l], [0], [3
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S1 | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_gw_redirect | grep cr-DR-S2 | wc -l], [0], [1
> +])
> +AT_CLEANUP
> +
> +AT_SETUP([ovn-northd -- lr multiple gw ports NAT])
> +ovn_start
> +
> +# Logical network:
> +# 1 LR, 3 Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# We will validate basic SNAT, DNAT and DNAT_AND_SNAT with
> +# multiple distributed gateway LRPs.
> +ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> +ovn-sbctl chassis-add gw2 geneve 128.0.0.1
> +ovn-sbctl chassis-add gw3 geneve 129.0.0.1
> +
> +ovn-nbctl lr-add DR
> +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24
> +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24
> +
> +ovn-nbctl ls-add S1
> +ovn-nbctl lsp-add S1 S1-DR
> +ovn-nbctl lsp-set-type S1-DR router
> +ovn-nbctl lsp-set-addresses S1-DR router
> +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +
> +ovn-nbctl ls-add S2
> +ovn-nbctl lsp-add S2 S2-DR
> +ovn-nbctl lsp-set-type S2-DR router
> +ovn-nbctl lsp-set-addresses S2-DR router
> +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +
> +ovn-nbctl ls-add S3
> +ovn-nbctl lsp-add S3 S3-DR
> +ovn-nbctl lsp-set-type S3-DR router
> +ovn-nbctl lsp-set-addresses S3-DR router
> +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +
> +ovn-nbctl ls-add ls
> +ovn-nbctl lsp-add ls ls-DR
> +ovn-nbctl lsp-set-type ls-DR router
> +ovn-nbctl lsp-set-addresses ls-DR router
> +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
> +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
> +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
> +
> +ovn-nbctl --wait=sb sync
> +
> +# Configure SNAT
> +ovn-nbctl lr-nat-add DR snat 172.16.1.1 20.0.0.10
> +ovn-nbctl lr-nat-add DR snat 10.0.0.1 20.0.0.10
> +ovn-nbctl lr-nat-add DR snat 192.168.0.1 20.0.0.10
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.1" | grep cr-DR-S1 | grep ct_snat | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.1)"| wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.1" | grep cr-DR-S2 | grep ct_snat | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.1)"| wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.1" | grep cr-DR-S3 | grep ct_snat | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.1)"| wc -l], [0], [1
> +])
> +
> +ovn-nbctl lr-nat-del DR snat 20.0.0.10
> +ovn-nbctl lr-nat-del DR snat 20.0.0.10
> +ovn-nbctl lr-nat-del DR snat 20.0.0.10
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0
> +])
> +
> +# Configure DNAT
> +ovn-nbctl lr-nat-add DR dnat 172.16.1.10 20.0.0.10
> +ovn-nbctl lr-nat-add DR dnat 10.0.0.10 20.0.0.10
> +ovn-nbctl lr-nat-add DR dnat 192.168.0.10 20.0.0.10
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1
> +])
> +
> +ovn-nbctl lr-nat-del DR dnat 172.16.1.10
> +ovn-nbctl lr-nat-del DR dnat 10.0.0.10
> +ovn-nbctl lr-nat-del DR dnat 192.168.0.10
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0
> +])
> +
> +# Configure DNAT_AND_SNAT
> +ovn-nbctl lr-nat-add DR dnat_and_snat 172.16.1.10 20.0.0.10
> +ovn-nbctl lr-nat-add DR dnat_and_snat 10.0.0.10 20.0.0.10
> +ovn-nbctl lr-nat-add DR dnat_and_snat 192.168.0.10 20.0.0.10
> +
> +ovn-sbctl dump-flows DR
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [3
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [3
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep ct_snat | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep "ct_snat(172.16.1.10)"| wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep ct_snat | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep "ct_snat(10.0.0.10)"| wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep ct_snat | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep "ct_snat(192.168.0.10)"| wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat| wc -l], [0], [3
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat| wc -l], [0], [3
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 172.16.1.10" | grep cr-DR-S1 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S1 | grep ct_dnat | wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 10.0.0.10" | grep cr-DR-S2 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S2 | grep ct_dnat | wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep "ip4.dst == 192.168.0.10" | grep cr-DR-S3 | grep "ct_dnat(20.0.0.10)" | wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep "ip4.src == 20.0.0.10" | grep cr-DR-S3 | grep ct_dnat | wc -l], [0], [1
> +])
> +
> +ovn-nbctl lr-nat-del DR dnat_and_snat 172.16.1.10
> +ovn-nbctl lr-nat-del DR dnat_and_snat 10.0.0.10
> +ovn-nbctl lr-nat-del DR dnat_and_snat 192.168.0.10
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_unsnat | grep ct_snat| wc -l], [0], [0
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_snat | grep ct_snat| wc -l], [0], [0
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct_dnat | wc -l], [0], [0
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep ct_dnat | wc -l], [0], [0
> +])
> +AT_CLEANUP
> +
> +AT_SETUP([ovn-northd -- lr multiple gw ports LB])
> +ovn_start
> +
> +# Logical network:
> +# 1 LR, 3 Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# We will validate LB with multiple distributed gateway LRPs.
> +ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> +ovn-sbctl chassis-add gw2 geneve 128.0.0.1
> +ovn-sbctl chassis-add gw3 geneve 129.0.0.1
> +
> +ovn-nbctl lr-add DR
> +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 10.0.0.1/24
> +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> +ovn-nbctl lrp-add DR DR-ls 04:ac:10:01:00:01 20.0.0.0/24
> +
> +ovn-nbctl ls-add S1
> +ovn-nbctl lsp-add S1 S1-DR
> +ovn-nbctl lsp-set-type S1-DR router
> +ovn-nbctl lsp-set-addresses S1-DR router
> +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +
> +ovn-nbctl ls-add S2
> +ovn-nbctl lsp-add S2 S2-DR
> +ovn-nbctl lsp-set-type S2-DR router
> +ovn-nbctl lsp-set-addresses S2-DR router
> +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +
> +ovn-nbctl ls-add S3
> +ovn-nbctl lsp-add S3 S3-DR
> +ovn-nbctl lsp-set-type S3-DR router
> +ovn-nbctl lsp-set-addresses S3-DR router
> +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +
> +ovn-nbctl ls-add ls
> +ovn-nbctl lsp-add ls ls-DR
> +ovn-nbctl lsp-set-type ls-DR router
> +ovn-nbctl lsp-set-addresses ls-DR router
> +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
> +ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
> +ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
> +
> +ovn-nbctl --wait=sb sync
> +
> +ovn-nbctl lb-add lb0 192.168.0.3:80 10.0.0.2:80,10.0.0.3:80
> +ovn-nbctl lr-lb-add DR lb0
> +
> +ovn-sbctl dump-flows DR
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| wc -l], [0], [3
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| wc -l], [0], [3
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S1 | grep cr-DR-S1| wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S1| grep cr-DR-S1 | wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S2 | grep cr-DR-S2| wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S2| grep cr-DR-S2 | wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.est| grep DR-S3 | grep cr-DR-S3| wc -l], [0], [1
> +])
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_in_dnat | grep ct.new| grep DR-S3| grep cr-DR-S3 | wc -l], [0], [1
> +])
> +
> +AT_CHECK([ovn-sbctl dump-flows DR | grep lr_out_undnat | grep priority=120| wc -l], [0], [3
> +])
> +
> +AT_CLEANUP
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 9bac94b..7c44cd3 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -19691,7 +19691,7 @@ AT_CAPTURE_FILE([sbflows2])
> OVS_WAIT_FOR_OUTPUT(
> [ovn-sbctl dump-flows > sbflows2
> ovn-sbctl dump-flows lr0 | grep ct_lb | grep priority=120 | sed 's/table=..//'], 0,
> - [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");)
> + [ (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_lb(backends=10.0.0.3:80,20.0.0.3:80; hash_fields="ip_dst,ip_src,tcp_dst,tcp_src");)
> ])
>
> # get the svc monitor mac.
> @@ -19732,8 +19732,8 @@ AT_CHECK(
> AT_CAPTURE_FILE([sbflows4])
> ovn-sbctl dump-flows lr0 > sbflows4
> AT_CHECK([grep lr_in_dnat sbflows4 | grep priority=120 | sed 's/table=..//' | sort], [0], [dnl
> - (lr_in_dnat ), priority=120 , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> - (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && is_chassis_resident("cr-lr0-public")), action=(drop;)
> + (lr_in_dnat ), priority=120 , match=(ct.est && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_dnat;)
> + (lr_in_dnat ), priority=120 , match=(ct.new && ip && ip4.dst == 10.0.0.10 && tcp && tcp.dst == 80 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(drop;)
> ])
>
> # Delete sw0-p1
> @@ -23715,3 +23715,307 @@ OVS_WAIT_UNTIL([test `ovs-vsctl get Interface lsp2 external_ids:ovn-installed` =
>
> OVN_CLEANUP([hv1])
> AT_CLEANUP
> +
> +AT_SETUP([ovn -- lr multiple gw ports])
> +ovn_start
> +
> +# Logical network:
> +# 1 LR, 3 Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# S1 - VLAN 1000
> +# S2 - VLAN 2000
> +# S3 - VLAN 3000
> +#
> +# 5 chassis(s), HV1----HV5
> +#
> +# HV1 - VIF11
> +# HV2 - Gateway chassis gw1
> +# HV3 - Gateway chassis gw2
> +# HV4 - Gateway chassis gw3
> +# HV5 - North endpoint
> +
> +ovn-nbctl lr-add DR
> +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add DR DR-S2 08:ac:10:01:00:01 10.0.0.1/24
> +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> +ovn-nbctl lrp-add DR DR-ls 06:ac:10:01:00:01 20.0.0.0/24
> +
> +ovn-nbctl ls-add S1
> +ovn-nbctl lsp-add S1 S1-DR
> +ovn-nbctl lsp-set-type S1-DR router
> +ovn-nbctl lsp-set-addresses S1-DR router
> +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +ovn-nbctl lsp-add S1 ln1 "" 1000
> +ovn-nbctl lsp-set-addresses ln1 unknown
> +ovn-nbctl lsp-set-type ln1 localnet
> +ovn-nbctl lsp-set-options ln1 network_name=phys
> +
> +ovn-nbctl ls-add S2
> +ovn-nbctl lsp-add S2 S2-DR
> +ovn-nbctl lsp-set-type S2-DR router
> +ovn-nbctl lsp-set-addresses S2-DR router
> +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +ovn-nbctl lsp-add S2 ln2 "" 2000
> +ovn-nbctl lsp-set-addresses ln2 unknown
> +ovn-nbctl lsp-set-type ln2 localnet
> +ovn-nbctl lsp-set-options ln2 network_name=phys
> +
> +ovn-nbctl ls-add S3
> +ovn-nbctl lsp-add S3 S3-DR
> +ovn-nbctl lsp-set-type S3-DR router
> +ovn-nbctl lsp-set-addresses S3-DR router
> +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +ovn-nbctl lsp-add S3 ln3 "" 3000
> +ovn-nbctl lsp-set-addresses ln3 unknown
> +ovn-nbctl lsp-set-type ln3 localnet
> +ovn-nbctl lsp-set-options ln3 network_name=phys
> +
> +ovn-nbctl ls-add ls
> +ovn-nbctl lsp-add ls ls-DR
> +ovn-nbctl lsp-set-type ls-DR router
> +ovn-nbctl lsp-set-addresses ls-DR router
> +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +# Add the lsp lp11 to ls. This will map to VIF11.
> +ovn-nbctl lsp-add ls lp11
> +ovn-nbctl lsp-set-addresses lp11 "f0:00:00:00:00:10 20.0.0.10"
> +ovn-nbctl lsp-set-port-security lp11 f0:00:00:00:00:10
> +
> +# Add the Northbound endpoint, lp-north1
> +ovn-nbctl ls-add ls-north1
> +ovn-nbctl lsp-add ls-north1 ln4 "" 1000
> +ovn-nbctl lsp-set-addresses ln4 unknown
> +ovn-nbctl lsp-set-type ln4 localnet
> +ovn-nbctl lsp-set-options ln4 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north1 lp-north1
> +ovn-nbctl lsp-set-addresses lp-north1 "f0:f0:00:00:00:11 172.16.1.10"
> +ovn-nbctl lsp-set-port-security lp-north1 f0:f0:00:00:00:11
> +
> +# Add the Northbound endpoint, lp-north2
> +ovn-nbctl ls-add ls-north2
> +ovn-nbctl lsp-add ls-north2 ln5 "" 2000
> +ovn-nbctl lsp-set-addresses ln5 unknown
> +ovn-nbctl lsp-set-type ln5 localnet
> +ovn-nbctl lsp-set-options ln5 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north2 lp-north2
> +ovn-nbctl lsp-set-addresses lp-north2 "f0:f0:00:00:00:22 10.0.0.10"
> +ovn-nbctl lsp-set-port-security lp-north2 f0:f0:00:00:00:22
> +
> +# Add the Northbound endpoint, lp-north3
> +ovn-nbctl ls-add ls-north3
> +ovn-nbctl lsp-add ls-north3 ln6 "" 3000
> +ovn-nbctl lsp-set-addresses ln6 unknown
> +ovn-nbctl lsp-set-type ln6 localnet
> +ovn-nbctl lsp-set-options ln6 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north3 lp-north3
> +ovn-nbctl lsp-set-addresses lp-north3 "f0:f0:00:00:00:33 192.168.0.10"
> +ovn-nbctl lsp-set-port-security lp-north3 f0:f0:00:00:00:33
> +
> +# Add 5 chassis
> +net_add n1
> +for i in 1 2 3 4 5; do
> + sim_add hv$i
> + as hv$i
> + ovs-vsctl add-br br-phys
> + ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
> + ovn_attach n1 br-phys 192.168.0.$i 24 $encap
> +done
> +
> +# Add a vif on HV1
> +as hv1 ovs-vsctl add-port br-int vif11 -- \
> + set Interface vif11 external-ids:iface-id=lp11 \
> + options:tx_pcap=hv1/vif11-tx.pcap \
> + options:rxq_pcap=hv1/vif11-rx.pcap \
> + ofport-request=11
> +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp11` = xup])
> +
> +as hv5 ovs-vsctl add-port br-int vif-north1 -- \
> + set Interface vif-north1 external-ids:iface-id=lp-north1 \
> + options:tx_pcap=hv5/vif-north1-tx.pcap \
> + options:rxq_pcap=hv5/vif-north1-rx.pcap \
> + ofport-request=44
> +
> +as hv5 ovs-vsctl add-port br-int vif-north2 -- \
> + set Interface vif-north2 external-ids:iface-id=lp-north2 \
> + options:tx_pcap=hv5/vif-north2-tx.pcap \
> + options:rxq_pcap=hv5/vif-north2-rx.pcap \
> + ofport-request=45
> +
> +as hv5 ovs-vsctl add-port br-int vif-north3 -- \
> + set Interface vif-north3 external-ids:iface-id=lp-north3 \
> + options:tx_pcap=hv5/vif-north3-tx.pcap \
> + options:rxq_pcap=hv5/vif-north3-rx.pcap \
> + ofport-request=46
> +
> +ovn-nbctl lrp-set-gateway-chassis DR-S1 hv2
> +ovn-nbctl lrp-set-gateway-chassis DR-S2 hv3
> +ovn-nbctl lrp-set-gateway-chassis DR-S3 hv4
> +
> +ovn-nbctl --wait=sb sync
> +OVN_POPULATE_ARP
> +
> +vif_to_ls () {
> + case ${1} in dnl (
> + vif?[[11]]) echo ls ;; dnl (
> + vif-north1) echo ls-north1 ;; dnl (
> + vif-north2) echo ls-north2 ;; dnl (
> + vif-north3) echo ls-north3 ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +}
> +
> +vif_to_hv () {
> + case ${1} in dnl (
> + vif[[1]]?) echo hv1 ;; dnl (
> + vif-north1) echo hv5 ;; dnl (
> + vif-north2) echo hv5 ;; dnl (
> + vif-north3) echo hv5 ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +}
> +
> +vif_to_lrp () {
> + case ${1} in dnl (
> + vif?[[11]]) echo DR-ls ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +
> +}
> +
> +ip_to_hex() {
> + printf "%02x%02x%02x%02x" "${@}"
> +}
> +
> +# test_arp INPORT SHA SPA TPA
> +#
> +# Causes a packet to be received on INPORT. The packet is an ARP
> +# request with SHA, SPA, and TPA as specified.
> +test_arp() {
> + local inport=$1 sha=$2 spa=$3 tpa=$4
> + local request=ffffffffffff${sha}08060001080006040001${sha}${spa}ffffffffffff${tpa}
> + hv=`vif_to_hv $inport`
> + as $hv ovs-appctl netdev-dummy/receive $inport $request
> +}
> +
> +
> +test_ip() {
> + # This packet has bad checksums but logical L3 routing doesn't check.
> + local inport=${1} src_mac=${2} dst_mac=${3} src_ip=${4} dst_ip=${5} outport=${6}
> + local packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000
> + shift; shift; shift; shift; shift
> + hv=`vif_to_hv $inport`
> + as $hv ovs-appctl netdev-dummy/receive $inport $packet
> + in_ls=`vif_to_ls $inport`
> + for outport; do
> + out_ls=`vif_to_ls $outport`
> + if test $in_ls = $out_ls; then
> + # Ports on the same logical switch receive exactly the same packet.
> + echo $packet
> + else
> + # Routing decrements TTL and updates source and dest MAC
> + # (and checksum).
> + # For North-South, packet will come via gateway chassis, i.e hv3
> + if test $inport = vif-north1; then
> + echo f0000000001006ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + if test $outport = vif-north1; then
> + echo f0f00000001102ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + if test $outport = vif-north2; then
> + echo f0f00000002208ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + if test $outport = vif-north3; then
> + echo f0f00000003304ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + fi >> $outport.expected
> + done
> +}
> +
> +echo "------ OVN dump ------"
> +ovn-nbctl show
> +ovn-sbctl show
> +ovn-sbctl list port_binding
> +ovn-sbctl list mac_binding
> +ovn-sbctl list datapath_binding
> +
> +ovn-sbctl dump-flows DR
> +ovn-sbctl dump-flows S1
> +ovn-sbctl dump-flows ls
> +
> +echo "------ hv1 dump ------"
> +as hv1 ovs-vsctl show
> +as hv1 ovs-vsctl list Open_Vswitch
> +as hv1 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv2 dump ------"
> +as hv2 ovs-vsctl show
> +as hv2 ovs-vsctl list Open_Vswitch
> +as hv2 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv3 dump ------"
> +as hv3 ovs-vsctl show
> +as hv3 ovs-vsctl list Open_Vswitch
> +as hv3 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv4 dump ------"
> +as hv4 ovs-vsctl show
> +as hv4 ovs-vsctl list Open_Vswitch
> +as hv5 ovs-ofctl dump-flows br-int
> +
> +# N-S with lp-north1
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 172 16 1 10`
> +tip=`ip_to_hex 172 16 1 50`
> +test_arp vif-north1 f0f000000011 $sip $tip
> +
> +echo "Send traffic North to South"
> +sip=`ip_to_hex 172 16 1 10`
> +dip=`ip_to_hex 20 0 0 10`
> +test_ip vif-north1 f0f000000011 02ac10010001 $sip $dip vif11
> +# Confirm that North to south traffic works fine.
> +OVN_CHECK_PACKETS([hv1/vif11-tx.pcap], [vif11.expected])
> +
> +echo "Send traffic South to North"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 172 16 1 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north1
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS([hv5/vif-north1-tx.pcap], [vif-north1.expected])
> +
> +# N-S with lp-north2
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 10 0 0 10`
> +tip=`ip_to_hex 10 0 0 50`
> +test_arp vif-north2 f0f000000022 $sip $tip
> +
> +echo "Send traffic South to North"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 10 0 0 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north2
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS([hv5/vif-north2-tx.pcap], [vif-north2.expected])
> +
> +# N-S with lp-north3
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 192 168 0 10`
> +tip=`ip_to_hex 192 168 0 50`
> +test_arp vif-north3 f0f000000033 $sip $tip
> +
> +echo "Send traffic South to North"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 192 168 0 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north3
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS([hv5/vif-north3-tx.pcap], [vif-north3.expected])
> +
> +AT_CLEANUP
> diff --git a/utilities/ovn-nbctl.c b/utilities/ovn-nbctl.c
> index d67f5c4..912ede7 100644
> --- a/utilities/ovn-nbctl.c
> +++ b/utilities/ovn-nbctl.c
> @@ -4208,6 +4208,27 @@ done:
> return ret;
> }
>
> +static bool
> +is_nat_rule_conflict(const struct nbrec_logical_router *lr)
> +{
> + int num_l3dgw_ports = 0;
> +
> + /* TODO: Add a proper validation to confirm that multiple
> + * external ips for a logical ip do not belong to same router port. */
> + for (size_t i = 0; i < lr->n_ports; i++) {
> + const struct nbrec_logical_router_port *lrp = lr->ports[i];
> + if (lrp->n_gateway_chassis) {
> + num_l3dgw_ports++;
> + }
> + }
> +
> + if (num_l3dgw_ports > 1) {
> + return false;
> + }
> +
> + return true;
> +}
> +
> static void
> nbctl_lr_nat_add(struct ctl_context *ctx)
> {
> @@ -4369,12 +4390,14 @@ nbctl_lr_nat_add(struct ctl_context *ctx)
> should_return = true;
> }
> } else {
> - ctl_error(ctx, "a NAT with this type (%s) and %s (%s) "
> - "already exists",
> - nat_type,
> - is_snat ? "logical_ip" : "external_ip",
> - is_snat ? new_logical_ip : new_external_ip);
> - should_return = true;
> + if (is_nat_rule_conflict(lr)) {
> + ctl_error(ctx, "a NAT with this type (%s) and %s (%s) "
> + "already exists",
> + nat_type,
> + is_snat ? "logical_ip" : "external_ip",
> + is_snat ? new_logical_ip : new_external_ip);
> + should_return = true;
> + }
> }
> }
> }
> --
> 1.8.3.1
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
More information about the dev
mailing list