[ovs-dev] [PATCH ovn v2 2/2] ovn-northd: Multiple distributed gateway port support.
Numan Siddique
numans at ovn.org
Fri Aug 6 21:13:43 UTC 2021
On Thu, Aug 5, 2021 at 11:40 AM Han Zhou <hzhou at ovn.org> wrote:
>
> From: Ankur Sharma <ankurmnnit2004 at gmail.com>
>
> By default, OVN support only one DGP (distributed gateway port) per
> logical router. While a single DGP port suffices for most of the North
> South connectivity, there are requirements where a logical router could
> be connected to multiple external networks and based on routing decision
> packet could go to different ones.
>
> This patch adds flexibility of having multiple DGPs per logical router.
>
> Changes can classified as following:
> a. Data structure changes to allow multiple DGPs per ovn_datapath.
>
> b. Consumption of new data structure in logical flows for
> individual features.
>
> c. Features that require changes are:
> i. Regular NS traffic flow.
> ii. Network Address Translation.
> iii. Load Balancer
> iv. Gateway_mtu.
> v. reside-on-redirect-chassis
> vi. Misc code sections that assumed a single DGP.
>
> d. Except for reside-on-redirect-chassis all the other features
> could be extended to multiple DGPs. Reside on redirect
> chassis with its current specification could not be extended
> and hence should be used only with the logical router that
> has a single DGP.
>
> This patch doesn't support NAT & load-balancer features for multiple
> DGPs yet, but added validations that disables NAT/load-balancer
> features when there are more than one DGP configured per router.
>
> Signed-off-by: Ankur Sharma <ankurmnnit2004 at gmail.com>
> Co-authored-by: Dhathri Purohith <dhathri.purohith at nutanix.com>
> Signed-off-by: Dhathri Purohith <dhathri.purohith at nutanix.com>
> Co-authored-by: Abhiram Sangana <sangana.abhiram at nutanix.com>
> Signed-off-by: Abhiram Sangana <sangana.abhiram at nutanix.com>
> Co-authored-by: Han Zhou <hzhou at ovn.org>
> Signed-off-by: Han Zhou <hzhou at ovn.org>
Hi Han,
Thanks for v2. I did some testing with this patch in a simple 2 node
setup (using ovn-fake-multinode)
Below are the logical resources created
-----------------------------
[root at ovn-central ~]# ovn-nbctl show
switch 3a3c2522-fcce-49e5-8334-8a72547e7da6 (sw0)
port sw0-port4
addresses: ["50:54:00:00:00:06 dynamic"]
port sw0-port1
addresses: ["50:54:00:00:00:03 10.0.0.3 1000::3"]
port sw0-port2
addresses: ["50:54:00:00:00:04 10.0.0.4 1000::4"]
port sw0-lr0
type: router
router-port: lr0-sw0
port sw0-port3
addresses: ["50:54:00:00:00:05 dynamic"]
switch 0573bbd7-fca7-4f06-84ac-1939f879fd5f (sw1)
port sw1-lr0
type: router
router-port: lr0-sw1
port sw1-port1
addresses: ["40:54:00:00:00:03 20.0.0.3 2000::3"]
router c7cd8dab-4e6d-45f3-a8f4-3653d25ab476 (lr0)
port lr0-sw1
mac: "00:00:00:00:ff:02"
networks: ["20.0.0.1/24", "2000::a/64"]
port lr0-sw0
mac: "00:00:00:00:ff:01"
networks: ["10.0.0.1/24", "1000::a/64"]
gateway chassis: [ovn-chassis-1 ovn-chassis-2]
[root at ovn-central ~]#
[root at ovn-central ~]#
[root at ovn-central ~]# ovn-sbctl show
Chassis ovn-chassis-1
hostname: ovn-chassis-1
Encap geneve
ip: "170.168.0.4"
options: {csum="true"}
Port_Binding sw0-port1
Port_Binding sw0-port3
Chassis ovn-gw-1
hostname: ovn-gw-1
Encap geneve
ip: "170.168.0.3"
options: {csum="true"}
Chassis ovn-chassis-2
hostname: ovn-chassis-2
Encap geneve
ip: "170.168.0.5"
options: {csum="true"}
Port_Binding sw1-port1
Port_Binding sw0-port4
Port_Binding cr-lr0-sw0
---------
As you can see the logical router port lr0-sw0 is a distributed gw
port scheduled on chassis - ovn-chassis-2.
The issue I see is : I'm not able to ping from sw0-port1 (10.0.0.3,
claimed by chassis ovn-chassis-1) to 10.0.0.1 and I'm not able to ping
to sw1-port1 (20.0.0.3).
I'm able to ping the same from sw0-port4 to 10.0.0.1 and 20.0.0.3
(claimed by chassis ovn-chassis-2).
If I move cr-lr0-sw0 to ovn-chassis-1, then ping from sw0-port1 works
but not from sw0-port4.
Is this expected ?
Does it mean that if a logical router has multiple gateway router
ports connecting to geneve logical switches, then all the logical
ports of those switches
should be only bound on the corresponding gateway chassis ? I
understand this is the case with ovn-k8s.
If this is a restriction, I think we should document it.
Otherwise the patch LGTM and I'm fine with the feature. The only
issue I see is that of semantics. If router port is a gateway router
port, then its
a peer's logical switch is ideally expected to have a localnet port.
But in the case of ovn-k8s, that's not the case. It is for this
reason I thought
pinning the logical switch to a particular chassis could be better.
If you think having multiple gw router ports is better, then I'd
suggest documenting the limitations I mentioned above. Also please
see
a small nit below. And you can consider my Ack with these addressed.
Acked-by: Numan Siddique <numans at ovn.org>
Thanks
Numan
Thanks
Numan
> ---
> NEWS | 3 +
> northd/lrouter.dl | 103 +++++-------
> northd/ovn-northd.8.xml | 6 +-
> northd/ovn-northd.c | 356 +++++++++++++++++++++++-----------------
> northd/ovn_northd.dl | 169 ++++++++++---------
> ovn-architecture.7.xml | 19 ++-
> ovn-nb.xml | 27 ++-
> tests/ovn-northd.at | 82 +++++++++
> tests/ovn.at | 307 ++++++++++++++++++++++++++++++++++
> 9 files changed, 771 insertions(+), 301 deletions(-)
>
> diff --git a/NEWS b/NEWS
> index f328666da..9f701caa7 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -35,6 +35,9 @@ OVN v21.06.0 - 18 Jun 2021
> "ovn-trim-limit-lflow-cache" and "ovn-trim-wmark-perc-lflow-cache", to
> allow enforcing a lflow cache size limit and high watermark percentage
> for which automatic memory trimming is performed.
> + - Support multiple distributed gateway ports on a single logical router.
> + (NAT and load-balancer are not supported yet when there are multiple
> + distributed gateway ports).
>
> OVN v21.03.0 - 12 Mar 2021
> -------------------------
> diff --git a/northd/lrouter.dl b/northd/lrouter.dl
> index 4a24f3f61..d37350ab8 100644
> --- a/northd/lrouter.dl
> +++ b/northd/lrouter.dl
> @@ -138,14 +138,14 @@ Warning[message] :-
> var message = "Bad configuration: distributed gateway port configured on "
> "port ${lrp.name} on L3 gateway router".
>
> -/* DistributedGatewayPortCandidate.
> +/* Distributed gateway ports.
> *
> - * Each row pairs a logical router with its distributed gateway port,
> - * but without checking that there is at most one DGP per LR.
> + * Each row means 'lrp' is a distributed gateway port on 'lr_uuid'.
> *
> - * (Use DistributedGatewayPort instead, since it guarantees uniqueness.) */
> -relation DistributedGatewayPortCandidate(lr_uuid: uuid, lrp_uuid: uuid)
> -DistributedGatewayPortCandidate(lr_uuid, lrp_uuid) :-
> + * A logical router can have multiple distributed gateway ports. */
> +relation DistributedGatewayPort(lrp: Intern<nb::Logical_Router_Port>,
> + lr_uuid: uuid)
> +DistributedGatewayPort(lrp, lr_uuid) :-
> lr in nb::Logical_Router(._uuid = lr_uuid),
> LogicalRouterPort(lrp_uuid, lr._uuid),
> lrp in &nb::Logical_Router_Port(._uuid = lrp_uuid),
> @@ -153,30 +153,10 @@ DistributedGatewayPortCandidate(lr_uuid, lrp_uuid) :-
> var has_hcg = lrp.ha_chassis_group.is_some(),
> var has_gc = not lrp.gateway_chassis.is_empty(),
> has_hcg or has_gc.
> -Warning[message] :-
> - DistributedGatewayPortCandidate(lr_uuid, lrp_uuid),
> - var lrps = lrp_uuid.group_by(lr_uuid).to_set(),
> - lrps.size() > 1,
> - lr in nb::Logical_Router(._uuid = lr_uuid),
> - var message = "Bad configuration: multiple distributed gateway ports on "
> - "logical router ${lr.name}; ignoring all of them".
> -
> -/* Distributed gateway ports.
> - *
> - * Each row means 'lrp' is the distributed gateway port on 'lr_uuid'.
> - *
> - * There is at most one distributed gateway port per logical router. */
> -relation DistributedGatewayPort(lrp: Intern<nb::Logical_Router_Port>, lr_uuid: uuid)
> -DistributedGatewayPort(lrp, lr_uuid) :-
> - DistributedGatewayPortCandidate(lr_uuid, lrp_uuid),
> - var lrps = lrp_uuid.group_by(lr_uuid).to_set(),
> - lrps.size() == 1,
> - Some{var lrp_uuid} = lrps.nth(0),
> - lrp in &nb::Logical_Router_Port(._uuid = lrp_uuid).
>
> /* HAChassis is an abstraction over nb::Gateway_Chassis and nb::HA_Chassis, which
> * are different ways to represent the same configuration. Each row is
> - * effectively one HA_Chassis record. (Usually, we could associated each
> + * effectively one HA_Chassis record. (Usually, we could associate each
> * row with a particular 'lr_uuid', but it's permissible for more than one
> * logical router to use a HA chassis group, so we omit it so that multiple
> * references get merged.)
> @@ -236,18 +216,20 @@ HAChassisGroup(ha_chassis_group_uuid(hac_group_uuid),
> .name = name,
> .external_ids = external_ids).
>
> -/* Each row maps from a logical router to the name of its HAChassisGroup.
> - * This level of indirection is needed because multiple logical routers
> - * are allowed to reference a given HAChassisGroup. */
> -relation LogicalRouterHAChassisGroup(lr_uuid: uuid,
> - hacg_uuid: uuid)
> -LogicalRouterHAChassisGroup(lr_uuid, ha_chassis_group_uuid(lrp._uuid)) :-
> - DistributedGatewayPort(lrp, lr_uuid),
> +/* Each row maps from a distributed gateway logical router port to the name of
> + * its HAChassisGroup.
> + * This level of indirection is needed because multiple distributed gateway
> + * logical router ports are allowed to reference a given HAChassisGroup. */
> +relation DistributedGatewayPortHAChassisGroup(
> + lrp: Intern<nb::Logical_Router_Port>,
> + hacg_uuid: uuid)
> +DistributedGatewayPortHAChassisGroup(lrp, ha_chassis_group_uuid(lrp._uuid)) :-
> + DistributedGatewayPort(.lrp = lrp),
> lrp.ha_chassis_group == None,
> lrp.gateway_chassis.size() > 0.
> -LogicalRouterHAChassisGroup(lr_uuid,
> - ha_chassis_group_uuid(hac_group_uuid)) :-
> - DistributedGatewayPort(lrp, lr_uuid),
> +DistributedGatewayPortHAChassisGroup(lrp,
> + ha_chassis_group_uuid(hac_group_uuid)) :-
> + DistributedGatewayPort(.lrp = lrp),
> Some{var hac_group_uuid} = lrp.ha_chassis_group,
> nb::HA_Chassis_Group(._uuid = hac_group_uuid).
>
> @@ -259,14 +241,19 @@ RouterPortIsRedirect(lrp, false) :-
> &nb::Logical_Router_Port(._uuid = lrp),
> not DistributedGatewayPort(&nb::Logical_Router_Port{._uuid = lrp}, _).
>
> -relation LogicalRouterRedirectPort(lr: uuid, has_redirect_port: Option<Intern<nb::Logical_Router_Port>>)
> -
> -LogicalRouterRedirectPort(lr, Some{lrp}) :-
> - DistributedGatewayPort(lrp, lr).
> -
> -LogicalRouterRedirectPort(lr, None) :-
> - nb::Logical_Router(._uuid = lr),
> - not DistributedGatewayPort(_, lr).
> +/*
> + * LogicalRouterDGWPorts maps from each logical router UUID
> + * to the logical router's set of distributed gateway (or redirect) ports. */
> +relation LogicalRouterDGWPorts(
> + lr_uuid: uuid,
> + l3dgw_ports: Vec<Intern<nb::Logical_Router_Port>>)
> +LogicalRouterDGWPorts(lr_uuid, l3dgw_ports) :-
> + DistributedGatewayPort(lrp, lr_uuid),
> + var l3dgw_ports = lrp.group_by(lr_uuid).to_vec().
> +LogicalRouterDGWPorts(lr_uuid, vec_empty()) :-
> + lr in nb::Logical_Router(),
> + var lr_uuid = lr._uuid,
> + not DistributedGatewayPort(_, lr_uuid).
>
> typedef ExceptionalExtIps = AllowedExtIps{ips: Intern<nb::Address_Set>}
> | ExemptedExtIps{ips: Intern<nb::Address_Set>}
> @@ -450,9 +437,7 @@ LogicalRouterCopp0(lr, meters) :-
>
> /* Router relation collects all attributes of a logical router.
> *
> - * `l3dgw_port` - optional redirect port (see `DistributedGatewayPort`)
> - * `redirect_port_name` - derived redirect port name (or empty string if
> - * router does not have a redirect port)
> + * `l3dgw_ports` - optional redirect ports (see `DistributedGatewayPort`)
> * `is_gateway` - true iff the router is a gateway router. Together with
> * `l3dgw_port`, this flag affects the generation of various flows
> * related to NAT and load balancing.
> @@ -474,8 +459,7 @@ typedef Router = Router {
> external_ids: Map<string,string>,
>
> /* Additional computed fields. */
> - l3dgw_port: Option<Intern<nb::Logical_Router_Port>>,
> - redirect_port_name: string,
> + l3dgw_ports: Vec<Intern<nb::Logical_Router_Port>>,
> is_gateway: bool,
> nats: Vec<NAT>,
> snat_ips: Map<v46_ip, Set<NAT>>,
> @@ -498,23 +482,18 @@ Router[Router{
> .options = lr.options,
> .external_ids = lr.external_ids,
>
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name =
> - match (l3dgw_port) {
> - Some{rport} -> json_string_escape(chassis_redirect_name(rport.name)),
> - _ -> ""
> - },
> - .is_gateway = lr.options.contains_key("chassis"),
> - .nats = nats,
> - .snat_ips = snat_ips,
> - .lbs = lbs,
> - .mcast_cfg = mcast_cfg,
> + .l3dgw_ports = l3dgw_ports,
> + .is_gateway = lr.options.contains_key("chassis"),
> + .nats = nats,
> + .snat_ips = snat_ips,
> + .lbs = lbs,
> + .mcast_cfg = mcast_cfg,
> .learn_from_arp_request = learn_from_arp_request,
> .force_lb_snat = force_lb_snat,
> .copp = copp}.intern()] :-
> lr in nb::Logical_Router(),
> lr.is_enabled(),
> - LogicalRouterRedirectPort(lr._uuid, l3dgw_port),
> + LogicalRouterDGWPorts(lr._uuid, l3dgw_ports),
> LogicalRouterNATs(lr._uuid, nats),
> LogicalRouterLBs(lr._uuid, lbs),
> LogicalRouterSnatIPs(lr._uuid, snat_ips),
> diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml
> index de4fe90c7..5afae743f 100644
> --- a/northd/ovn-northd.8.xml
> +++ b/northd/ovn-northd.8.xml
> @@ -3815,10 +3815,10 @@ icmp6 {
> <h3>Ingress Table 17: Gateway Redirect</h3>
>
> <p>
> - For distributed logical routers where one of the logical router
> + For distributed logical routers where one or more of the logical router
> ports specifies a gateway chassis, this table redirects
> - certain packets to the distributed gateway port instance on the
> - gateway chassis. This table has the following flows:
> + certain packets to the distributed gateway port instances on the
> + gateway chassises. This table has the following flows:
> </p>
>
> <ul>
> diff --git a/northd/ovn-northd.c b/northd/ovn-northd.c
> index 605e33486..b7398004d 100644
> --- a/northd/ovn-northd.c
> +++ b/northd/ovn-northd.c
> @@ -655,13 +655,12 @@ struct ovn_datapath {
> bool is_gw_router;
>
> /* OVN northd only needs to know about the logical router gateway port for
> - * NAT on a distributed router. This "distributed gateway port" is
> - * populated only when there is a gateway chassis specified for one of
> - * the ports on the logical router. Otherwise this will be NULL. */
> - struct ovn_port *l3dgw_port;
> - /* The "derived" OVN port representing the instance of l3dgw_port on
> - * the gateway chassis. */
> - struct ovn_port *l3redirect_port;
> + * NAT on a distributed router. The "distributed gateway ports" are
> + * populated only when there is a gateway chassis or ha chassis group
> + * specified for some of the ports on the logical router. Otherwise this
> + * will be NULL. */
> + struct ovn_port **l3dgw_ports;
> + size_t n_l3dgw_ports;
>
> /* NAT entries configured on the router. */
> struct ovn_nat *nat_entries;
> @@ -802,6 +801,16 @@ init_nat_entries(struct ovn_datapath *od)
> return;
> }
>
> + if (od->n_l3dgw_ports > 1) {
> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> + VLOG_WARN_RL(&rl, "NAT is configured on logical router %s, which has %"
> + PRIuSIZE" distributed gateway ports. NAT is not supported"
> + " yet when there is more than one distributed gateway "
> + "port on the router.",
> + od->nbr->name, od->n_l3dgw_ports);
> + return;
> + }
> +
> od->nat_entries = xmalloc(od->nbr->n_nat * sizeof *od->nat_entries);
>
> for (size_t i = 0; i < od->nbr->n_nat; i++) {
> @@ -941,6 +950,7 @@ ovn_datapath_destroy(struct hmap *datapaths, struct ovn_datapath *od)
> destroy_lb_ips(od);
> free(od->nat_entries);
> free(od->localnet_ports);
> + free(od->l3dgw_ports);
> ovn_ls_port_group_destroy(&od->nb_pgs);
> destroy_mcast_info_for_datapath(od);
>
> @@ -1489,9 +1499,18 @@ struct ovn_port {
> /* Logical port multicast data. */
> struct mcast_port_info mcast_info;
>
> - /* This is ordinarily false. It is true if and only if this ovn_port is
> - * derived from a chassis-redirect port. */
> - bool derived;
> + /* At most one of l3dgw_port and cr_port can be not NULL. */
> +
> + /* This is set to a distributed gateway port if and only if this ovn_port
> + * is "derived" from it. Otherwise this is set to NULL. The derived
> + * ovn_port represents the instance of distributed gateway port on the
> + * gateway chassis.*/
> + struct ovn_port *l3dgw_port;
> +
> + /* This is set to the "derived" chassis-redirect port of this port if and
> + * only if this port is a distributed gateway port. Otherwise this is set
> + * to NULL. */
> + struct ovn_port *cr_port;
>
> bool has_unknown; /* If the addresses have 'unknown' defined. */
>
> @@ -1512,6 +1531,18 @@ struct ovn_port {
> struct ovs_list list; /* In list of similar records. */
> };
>
> +static bool
> +is_l3dgw_port(const struct ovn_port *op)
> +{
> + return op->cr_port;
Since the return type is bool, I'd suggest - return !!op->cr_port;
> +}
> +
> +static bool
> +is_cr_port(const struct ovn_port *op)
> +{
> + return op->l3dgw_port;
Same as above.
> +}
> +
> static void
> destroy_routable_addresses(struct ovn_port_routable_addresses *ra)
> {
> @@ -1578,7 +1609,7 @@ ovn_port_create(struct hmap *ports, const char *key,
> op->key = xstrdup(key);
> op->sb = sb;
> ovn_port_set_nb(op, nbsp, nbrp);
> - op->derived = false;
> + op->l3dgw_port = op->cr_port = NULL;
> hmap_insert(ports, &op->key_node, hash_string(op->key, 0));
> return op;
> }
> @@ -1682,7 +1713,7 @@ lrport_is_enabled(const struct nbrec_logical_router_port *lrport)
> static struct ovn_port *
> ovn_port_get_peer(struct hmap *ports, struct ovn_port *op)
> {
> - if (!op->nbsp || !lsp_is_router(op->nbsp) || op->derived) {
> + if (!op->nbsp || !lsp_is_router(op->nbsp) || op->l3dgw_port) {
> return NULL;
> }
>
> @@ -2426,6 +2457,7 @@ join_logical_ports(struct northd_context *ctx,
> tag_alloc_add_existing_tags(tag_alloc_table, nbsp);
> }
> } else {
> + size_t n_allocated_l3dgw_ports = 0;
> for (size_t i = 0; i < od->nbr->n_ports; i++) {
> const struct nbrec_logical_router_port *nbrp
> = od->nbr->ports[i];
> @@ -2481,36 +2513,32 @@ join_logical_ports(struct northd_context *ctx,
> "on L3 gateway router", nbrp->name);
> continue;
> }
> - if (od->l3dgw_port || od->l3redirect_port) {
> - static struct vlog_rate_limit rl
> - = VLOG_RATE_LIMIT_INIT(1, 1);
> - VLOG_WARN_RL(&rl, "Bad configuration: multiple "
> - "distributed gateway ports on logical "
> - "router %s", od->nbr->name);
> - continue;
> - }
>
> char *redirect_name =
> ovn_chassis_redirect_name(nbrp->name);
> struct ovn_port *crp = ovn_port_find(ports, redirect_name);
> if (crp && crp->sb && crp->sb->datapath == od->sb) {
> - crp->derived = true;
> ovn_port_set_nb(crp, NULL, nbrp);
> ovs_list_remove(&crp->list);
> ovs_list_push_back(both, &crp->list);
> } else {
> crp = ovn_port_create(ports, redirect_name,
> NULL, nbrp, NULL);
> - crp->derived = true;
> ovs_list_push_back(nb_only, &crp->list);
> }
> + crp->l3dgw_port = op;
> + op->cr_port = crp;
> crp->od = od;
> free(redirect_name);
>
> - /* Set l3dgw_port and l3redirect_port in od, for later
> - * use during flow creation. */
> - od->l3dgw_port = op;
> - od->l3redirect_port = crp;
> + /* Add to l3dgw_ports in od, for later use during flow
> + * creation. */
> + if (od->n_l3dgw_ports == n_allocated_l3dgw_ports) {
> + od->l3dgw_ports = x2nrealloc(od->l3dgw_ports,
> + &n_allocated_l3dgw_ports,
> + sizeof *od->l3dgw_ports);
> + }
> + od->l3dgw_ports[od->n_l3dgw_ports++] = op;
>
> assign_routable_addresses(op);
> }
> @@ -2522,7 +2550,7 @@ join_logical_ports(struct northd_context *ctx,
> * to their peers. */
> struct ovn_port *op;
> HMAP_FOR_EACH (op, key_node, ports) {
> - if (op->nbsp && lsp_is_router(op->nbsp) && !op->derived) {
> + if (op->nbsp && lsp_is_router(op->nbsp) && !op->l3dgw_port) {
> struct ovn_port *peer = ovn_port_get_peer(ports, op);
> if (!peer || !peer->nbrp) {
> continue;
> @@ -2553,7 +2581,7 @@ join_logical_ports(struct northd_context *ctx,
> if (peer->od && peer->od->mcast_info.rtr.relay) {
> op->od->mcast_info.sw.flood_relay = true;
> }
> - } else if (op->nbrp && op->nbrp->peer && !op->derived) {
> + } else if (op->nbrp && op->nbrp->peer && !op->l3dgw_port) {
> struct ovn_port *peer = ovn_port_find(ports, op->nbrp->peer);
> if (peer) {
> if (peer->nbrp) {
> @@ -2598,7 +2626,8 @@ get_nat_addresses(const struct ovn_port *op, size_t *n, bool routable_only)
> struct eth_addr mac;
> if (!op || !op->nbrp || !op->od || !op->od->nbr
> || (!op->od->nbr->n_nat && !op->od->nbr->n_load_balancer)
> - || !eth_addr_from_string(op->nbrp->mac, &mac)) {
> + || !eth_addr_from_string(op->nbrp->mac, &mac)
> + || op->od->n_l3dgw_ports > 1) {
> *n = n_nats;
> return NULL;
> }
> @@ -2629,7 +2658,7 @@ get_nat_addresses(const struct ovn_port *op, size_t *n, bool routable_only)
>
> /* Determine whether this NAT rule satisfies the conditions for
> * distributed NAT processing. */
> - if (op->od->l3redirect_port && !strcmp(nat->type, "dnat_and_snat")
> + if (op->od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat")
> && nat->logical_port && nat->external_mac) {
> /* Distributed NAT rule. */
> if (eth_addr_from_string(nat->external_mac, &mac)) {
> @@ -2695,9 +2724,9 @@ get_nat_addresses(const struct ovn_port *op, size_t *n, bool routable_only)
> if (central_ip_address) {
> /* Gratuitous ARP for centralized NAT rules on distributed gateway
> * ports should be restricted to the gateway chassis. */
> - if (op->od->l3redirect_port) {
> + if (op->od->n_l3dgw_ports) {
> ds_put_format(&c_addresses, " is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->od->l3dgw_ports[0]->cr_port->json_key);
> }
>
> addresses[n_nats++] = ds_steal_cstr(&c_addresses);
> @@ -3010,7 +3039,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> /* If the router is for l3 gateway, it resides on a chassis
> * and its port type is "l3gateway". */
> const char *chassis_name = smap_get(&op->od->nbr->options, "chassis");
> - if (op->derived) {
> + if (is_cr_port(op)) {
> sbrec_port_binding_set_type(op->sb, "chassisredirect");
> } else if (chassis_name) {
> sbrec_port_binding_set_type(op->sb, "l3gateway");
> @@ -3020,7 +3049,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
>
> struct smap new;
> smap_init(&new);
> - if (op->derived) {
> + if (is_cr_port(op)) {
> const char *redirect_type = smap_get(&op->nbrp->options,
> "redirect-type");
>
> @@ -3200,7 +3229,7 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> char **nats = NULL;
> if (nat_addresses && !strcmp(nat_addresses, "router")) {
> if (op->peer && op->peer->od
> - && (chassis || op->peer->od->l3redirect_port)) {
> + && (chassis || op->peer->od->n_l3dgw_ports)) {
> nats = get_nat_addresses(op->peer, &n_nats, false);
> }
> /* Only accept manual specification of ethernet address
> @@ -3236,12 +3265,26 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> * sending the GARPs for the router port IPs.
> * */
> bool add_router_port_garp = false;
> - if (op->peer && op->peer->nbrp && op->peer->od->l3dgw_port &&
> - op->peer->od->l3redirect_port &&
> - (smap_get_bool(&op->peer->nbrp->options,
> - "reside-on-redirect-chassis", false) ||
> - op->peer == op->peer->od->l3dgw_port)) {
> - add_router_port_garp = true;
> + if (op->peer && op->peer->nbrp && op->peer->od->n_l3dgw_ports) {
> + if (is_l3dgw_port(op->peer)) {
> + add_router_port_garp = true;
> + } else if (smap_get_bool(&op->peer->nbrp->options,
> + "reside-on-redirect-chassis", false)) {
> + if (op->peer->od->n_l3dgw_ports == 1) {
> + add_router_port_garp = true;
> + } else {
> + static struct vlog_rate_limit rl =
> + VLOG_RATE_LIMIT_INIT(1, 1);
> + VLOG_WARN_RL(&rl, "\"reside-on-redirect-chassis\" is "
> + "set on logical router port %s, which "
> + "is on logical router %s, which has %"
> + PRIuSIZE" distributed gateway ports. This"
> + "option can only be used when there is "
> + "a single distributed gateway port.",
> + op->peer->key, op->peer->od->nbr->name,
> + op->peer->od->n_l3dgw_ports);
> + }
> + }
> } else if (chassis && op->od->n_localnet_ports) {
> add_router_port_garp = true;
> }
> @@ -3256,9 +3299,10 @@ ovn_port_update_sbrec(struct northd_context *ctx,
> op->peer->lrp_networks.ipv4_addrs[i].addr_s);
> }
>
> - if (op->peer->od->l3redirect_port) {
> + if (op->peer->od->n_l3dgw_ports) {
> ds_put_format(&garp_info, " is_chassis_resident(%s)",
> - op->peer->od->l3redirect_port->json_key);
> + op->peer->od->l3dgw_ports[0]
> + ->cr_port->json_key);
> }
>
> n_nats++;
> @@ -3531,7 +3575,17 @@ build_ovn_lr_lbs(struct hmap *datapaths, struct hmap *lbs)
> if (!od->nbr) {
> continue;
> }
> - if (!smap_get(&od->nbr->options, "chassis") && !od->l3dgw_port) {
> + if (!smap_get(&od->nbr->options, "chassis")
> + && od->n_l3dgw_ports != 1) {
> + if (od->n_l3dgw_ports > 1 && od->nbr->n_load_balancer) {
> + static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(1, 1);
> + VLOG_WARN_RL(&rl, "Load-balancers are configured on logical "
> + "router %s, which has %"PRIuSIZE" distributed "
> + "gateway ports. Load-balancer is not supported "
> + "yet when there is more than one distributed "
> + "gateway port on the router.",
> + od->nbr->name, od->n_l3dgw_ports);
> + }
> continue;
> }
>
> @@ -6433,13 +6487,14 @@ build_lrouter_groups__(struct hmap *ports, struct ovn_datapath *od)
> {
> ovs_assert((od && od->nbr && od->lr_group));
>
> - if (od->l3dgw_port && od->l3redirect_port) {
> - /* It's a logical router with gateway port. If it
> - * has HA_Chassis_Group associated to it in SB DB, then store the
> - * ha chassis group name. */
> - if (od->l3redirect_port->sb->ha_chassis_group) {
> + /* For logical router with distributed gateway ports. If it
> + * has HA_Chassis_Group associated to it in SB DB, then store the
> + * ha chassis group name. */
> + for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
> + struct ovn_port *crp = od->l3dgw_ports[i]->cr_port;
> + if (crp->sb->ha_chassis_group) {
> sset_add(&od->lr_group->ha_chassis_groups,
> - od->l3redirect_port->sb->ha_chassis_group->name);
> + crp->sb->ha_chassis_group->name);
> }
> }
>
> @@ -7800,16 +7855,17 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
> ds_clear(match);
> ds_put_format(match, "eth.dst == "ETH_ADDR_FMT,
> ETH_ADDR_ARGS(mac));
> - if (op->peer->od->l3dgw_port
> - && op->peer->od->l3redirect_port
> + if (op->peer->od->n_l3dgw_ports
> && op->od->n_localnet_ports) {
> bool add_chassis_resident_check = false;
> - if (op->peer == op->peer->od->l3dgw_port) {
> + const char *json_key;
> + if (is_l3dgw_port(op->peer)) {
> /* The peer of this port represents a distributed
> * gateway port. The destination lookup flow for the
> * router's distributed gateway port MAC address should
> * only be programmed on the gateway chassis. */
> add_chassis_resident_check = true;
> + json_key = op->peer->cr_port->json_key;
> } else {
> /* Check if the option 'reside-on-redirect-chassis'
> * is set to true on the peer port. If set to true
> @@ -7820,12 +7876,15 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
> */
> add_chassis_resident_check = smap_get_bool(
> &op->peer->nbrp->options,
> - "reside-on-redirect-chassis", false);
> + "reside-on-redirect-chassis", false) &&
> + op->peer->od->n_l3dgw_ports == 1;
> + json_key =
> + op->peer->od->l3dgw_ports[0]->cr_port->json_key;
> }
>
> if (add_chassis_resident_check) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->peer->od->l3redirect_port->json_key);
> + json_key);
> }
> }
>
> @@ -7838,8 +7897,7 @@ build_lswitch_ip_unicast_lookup(struct ovn_port *op,
>
> /* Add ethernet addresses specified in NAT rules on
> * distributed logical routers. */
> - if (op->peer->od->l3dgw_port
> - && op->peer == op->peer->od->l3dgw_port) {
> + if (is_l3dgw_port(op->peer)) {
> for (int j = 0; j < op->peer->od->nbr->n_nat; j++) {
> const struct nbrec_nat *nat
> = op->peer->od->nbr->nat[j];
> @@ -9106,14 +9164,14 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> &lb->nlb->header_);
> }
>
> - if (od->l3redirect_port &&
> + if (od->n_l3dgw_ports &&
> (lb_vip->n_backends || !lb_vip->empty_backend_rej)) {
> new_match_p = xasprintf("%s && is_chassis_resident(%s)",
> new_match,
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> est_match_p = xasprintf("%s && is_chassis_resident(%s)",
> est_match,
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
>
> if (snat_type == NO_FORCE_SNAT &&
> @@ -9158,15 +9216,15 @@ build_lrouter_nat_flows_for_lb(struct ovn_lb_vip *lb_vip,
> free(est_match_p);
> }
>
> - if (!od->l3dgw_port || !od->l3redirect_port || !lb_vip->n_backends) {
> + if (!od->n_l3dgw_ports || !lb_vip->n_backends) {
> goto next;
> }
>
> - char *undnat_match_p = xasprintf("%s) && outport == %s && "
> - "is_chassis_resident(%s)",
> - ds_cstr(&undnat_match),
> - od->l3dgw_port->json_key,
> - od->l3redirect_port->json_key);
> + char *undnat_match_p = xasprintf(
> + "%s) && outport == %s && is_chassis_resident(%s)",
> + ds_cstr(&undnat_match),
> + od->l3dgw_ports[0]->json_key,
> + od->l3dgw_ports[0]->cr_port->json_key);
> if (snat_type == SKIP_SNAT) {
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_OUT_UNDNAT, 120,
> undnat_match_p, skip_snat_est_action,
> @@ -9662,9 +9720,9 @@ build_lrouter_port_nat_arp_nd_flow(struct ovn_port *op,
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ARP responses
> * from different chassis. */
> - if (op->od->l3redirect_port) {
> + if (op->od->n_l3dgw_ports) {
> ds_put_format(&match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->od->l3dgw_ports[0]->cr_port->json_key);
> }
> }
>
> @@ -9938,7 +9996,7 @@ build_adm_ctrl_flows_for_lrouter_port(
> return;
> }
>
> - if (op->derived) {
> + if (is_cr_port(op)) {
> /* No ingress packets should be received on a chassisredirect
> * port. */
> return;
> @@ -9963,12 +10021,11 @@ build_adm_ctrl_flows_for_lrouter_port(
> ds_clear(match);
> ds_put_format(match, "eth.dst == %s && inport == %s",
> op->lrp_networks.ea_s, op->json_key);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (is_l3dgw_port(op)) {
> /* Traffic with eth.dst = l3dgw_port->lrp_networks.ea_s
> * should only be received on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
> ovn_lflow_add_with_hint(lflows, op->od, S_ROUTER_IN_ADMISSION, 50,
> ds_cstr(match), ds_cstr(actions),
> @@ -10107,10 +10164,9 @@ build_neigh_learning_flows_for_lrouter_port(
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen,
> op->lrp_networks.ipv4_addrs[i].addr_s);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (is_l3dgw_port(op)) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
> const char *actions_s = REGBIT_LOOKUP_NEIGHBOR_RESULT
> " = lookup_arp(inport, arp.spa, arp.sha); "
> @@ -10127,10 +10183,9 @@ build_neigh_learning_flows_for_lrouter_port(
> op->json_key,
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (is_l3dgw_port(op)) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
> ds_clear(actions);
> ds_put_format(actions, REGBIT_LOOKUP_NEIGHBOR_RESULT
> @@ -10620,7 +10675,7 @@ build_arp_resolve_flows_for_lrouter_port(
> }
> }
>
> - if (!op->derived && op->od->l3redirect_port) {
> + if (is_l3dgw_port(op)) {
> const char *redirect_type = smap_get(&op->nbrp->options,
> "redirect-type");
> if (redirect_type && !strcasecmp(redirect_type, "bridged")) {
> @@ -10633,7 +10688,7 @@ build_arp_resolve_flows_for_lrouter_port(
> ds_clear(match);
> ds_put_format(match, "outport == %s && "
> "!is_chassis_resident(%s)", op->json_key,
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> ds_clear(actions);
> ds_put_format(actions, "eth.dst = %s; next;",
> op->lrp_networks.ea_s);
> @@ -10881,8 +10936,8 @@ build_arp_resolve_flows_for_lrouter_port(
> &op->nbsp->header_);
> }
>
> - if (smap_get(&peer->od->nbr->options, "chassis") ||
> - (peer->od->l3dgw_port && peer == peer->od->l3dgw_port)) {
> + if (smap_get(&peer->od->nbr->options, "chassis")
> + || peer->cr_port) {
> routable_addresses_to_lflows(lflows, router_port, peer,
> match, actions);
> }
> @@ -11079,32 +11134,32 @@ build_gateway_redirect_flows_for_lrouter(
> struct ovn_datapath *od, struct hmap *lflows,
> struct ds *match, struct ds *actions)
> {
> - if (od->nbr) {
> - if (od->l3dgw_port && od->l3redirect_port) {
> - const struct ovsdb_idl_row *stage_hint = NULL;
> -
> - if (od->l3dgw_port->nbrp) {
> - stage_hint = &od->l3dgw_port->nbrp->header_;
> - }
> + if (!od->nbr) {
> + return;
> + }
> + for (size_t i = 0; i < od->n_l3dgw_ports; i++) {
> + const struct ovsdb_idl_row *stage_hint = NULL;
>
> - /* For traffic with outport == l3dgw_port, if the
> - * packet did not match any higher priority redirect
> - * rule, then the traffic is redirected to the central
> - * instance of the l3dgw_port. */
> - ds_clear(match);
> - ds_put_format(match, "outport == %s",
> - od->l3dgw_port->json_key);
> - ds_clear(actions);
> - ds_put_format(actions, "outport = %s; next;",
> - od->l3redirect_port->json_key);
> - ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50,
> - ds_cstr(match), ds_cstr(actions),
> - stage_hint);
> + if (od->l3dgw_ports[i]->nbrp) {
> + stage_hint = &od->l3dgw_ports[i]->nbrp->header_;
> }
>
> - /* Packets are allowed by default. */
> - ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 0, "1", "next;");
> + /* For traffic with outport == l3dgw_port, if the
> + * packet did not match any higher priority redirect
> + * rule, then the traffic is redirected to the central
> + * instance of the l3dgw_port. */
> + ds_clear(match);
> + ds_put_format(match, "outport == %s",
> + od->l3dgw_ports[i]->json_key);
> + ds_clear(actions);
> + ds_put_format(actions, "outport = %s; next;",
> + od->l3dgw_ports[i]->cr_port->json_key);
> + ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_GW_REDIRECT, 50,
> + ds_cstr(match), ds_cstr(actions),
> + stage_hint);
> }
> + /* Packets are allowed by default. */
> + ovn_lflow_add(lflows, od, S_ROUTER_IN_GW_REDIRECT, 0, "1", "next;");
> }
>
> /* Local router ingress table ARP_REQUEST: ARP request.
> @@ -11203,7 +11258,7 @@ build_egress_delivery_flows_for_lrouter_port(
> return;
> }
>
> - if (op->derived) {
> + if (is_cr_port(op)) {
> /* No egress packets should be processed in the context of
> * a chassisredirect port. The chassisredirect port should
> * be replaced by the l3dgw port in the local output
> @@ -11293,7 +11348,7 @@ build_dhcpv6_reply_flows_for_lrouter_port(
> struct ovn_port *op, struct hmap *lflows,
> struct ds *match)
> {
> - if (op->nbrp && (!op->derived)) {
> + if (op->nbrp && (!op->l3dgw_port)) {
> for (size_t i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> ds_clear(match);
> ds_put_format(match, "ip6.dst == %s && udp.src == 547 &&"
> @@ -11313,7 +11368,7 @@ build_ipv6_input_flows_for_lrouter_port(
> struct ds *match, struct ds *actions,
> struct shash *meter_groups)
> {
> - if (op->nbrp && (!op->derived)) {
> + if (op->nbrp && (!op->l3dgw_port)) {
> /* No ingress packets are accepted on a chassisredirect
> * port, so no need to program flows for that port. */
> if (op->lrp_networks.n_ipv6_addrs) {
> @@ -11339,15 +11394,14 @@ build_ipv6_input_flows_for_lrouter_port(
> * router's own IP address. */
> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> ds_clear(match);
> - if (op->od->l3dgw_port && op == op->od->l3dgw_port
> - && op->od->l3redirect_port) {
> + if (is_l3dgw_port(op)) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
> * should only be sent from the gateway chassi, so that
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ND replies
> * from different chassis. */
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
>
> build_lrouter_nd_flow(op->od, op, "nd_na_router",
> @@ -11358,7 +11412,7 @@ build_ipv6_input_flows_for_lrouter_port(
> }
>
> /* UDP/TCP/SCTP port unreachable */
> - if (!op->od->is_gw_router && !op->od->l3dgw_port) {
> + if (!op->od->is_gw_router && !op->od->n_l3dgw_ports) {
> for (int i = 0; i < op->lrp_networks.n_ipv6_addrs; i++) {
> ds_clear(match);
> ds_put_format(match,
> @@ -11528,7 +11582,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> {
> /* No ingress packets are accepted on a chassisredirect
> * port, so no need to program flows for that port. */
> - if (op->nbrp && (!op->derived)) {
> + if (op->nbrp && (!op->l3dgw_port)) {
> if (op->lrp_networks.n_ipv4_addrs) {
> /* L3 admission control: drop packets that originate from an
> * IPv4 address owned by the router or a broadcast address
> @@ -11598,16 +11652,18 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> op->lrp_networks.ipv4_addrs[i].network_s,
> op->lrp_networks.ipv4_addrs[i].plen);
>
> - if (op->od->l3dgw_port && op->od->l3redirect_port && op->peer
> + if (op->od->n_l3dgw_ports && op->peer
> && op->peer->od->n_localnet_ports) {
> bool add_chassis_resident_check = false;
> - if (op == op->od->l3dgw_port) {
> + const char *json_key;
> + if (is_l3dgw_port(op)) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea_s
> * should only be sent from the gateway chassis, so that
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ARP responses
> * from different chassis. */
> add_chassis_resident_check = true;
> + json_key = op->cr_port->json_key;
> } else {
> /* Check if the option 'reside-on-redirect-chassis'
> * is set to true on the router port. If set to true
> @@ -11619,12 +11675,14 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> */
> add_chassis_resident_check = smap_get_bool(
> &op->nbrp->options,
> - "reside-on-redirect-chassis", false);
> + "reside-on-redirect-chassis", false) &&
> + op->od->n_l3dgw_ports == 1;
> + json_key = op->od->l3dgw_ports[0]->cr_port->json_key;
> }
>
> if (add_chassis_resident_check) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + json_key);
> }
> }
>
> @@ -11637,9 +11695,9 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> const char *ip_address;
> if (sset_count(&op->od->lb_ips_v4)) {
> ds_clear(match);
> - if (op == op->od->l3dgw_port) {
> + if (is_l3dgw_port(op)) {
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
>
> struct ds load_balancer_ips_v4 = DS_EMPTY_INITIALIZER;
> @@ -11657,9 +11715,9 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
>
> SSET_FOR_EACH (ip_address, &op->od->lb_ips_v6) {
> ds_clear(match);
> - if (op == op->od->l3dgw_port) {
> + if (is_l3dgw_port(op)) {
> ds_put_format(match, "is_chassis_resident(%s)",
> - op->od->l3redirect_port->json_key);
> + op->cr_port->json_key);
> }
>
> build_lrouter_nd_flow(op->od, op, "nd_na",
> @@ -11668,7 +11726,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> lflows, meter_groups);
> }
>
> - if (!op->od->is_gw_router && !op->od->l3dgw_port) {
> + if (!op->od->is_gw_router && !op->od->n_l3dgw_ports) {
> /* UDP/TCP/SCTP port unreachable. */
> for (int i = 0; i < op->lrp_networks.n_ipv4_addrs; i++) {
> ds_clear(match);
> @@ -11765,7 +11823,7 @@ build_lrouter_ipv4_ip_input(struct ovn_port *op,
> * exception is on the l3dgw_port where we might need to use a
> * different ETH address.
> */
> - if (op != op->od->l3dgw_port) {
> + if (!is_l3dgw_port(op)) {
> return;
> }
>
> @@ -11847,12 +11905,12 @@ build_lrouter_in_unsnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> ds_clear(actions);
> ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> is_v6 ? "6" : "4", nat->external_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
>
> if (!strcmp(nat->type, "dnat_and_snat") && stateless) {
> @@ -11924,12 +11982,12 @@ build_lrouter_in_dnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.dst == %s && inport == %s",
> is_v6 ? "6" : "4", nat->external_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
> ds_clear(actions);
> if (nat->allowed_ext_ips || nat->exempted_ext_ips) {
> @@ -11968,7 +12026,7 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> *
> * Note that this only applies for NAT on a distributed router.
> */
> - if (!od->l3dgw_port ||
> + if (!od->n_l3dgw_ports ||
> (strcmp(nat->type, "dnat") && strcmp(nat->type, "dnat_and_snat"))) {
> return;
> }
> @@ -11976,12 +12034,12 @@ build_lrouter_out_undnat_flow(struct hmap *lflows, struct ovn_datapath *od,
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> is_v6 ? "6" : "4", nat->logical_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
> ds_clear(actions);
> if (distributed) {
> @@ -12054,13 +12112,13 @@ build_lrouter_out_snat_flow(struct hmap *lflows, struct ovn_datapath *od,
> ds_clear(match);
> ds_put_format(match, "ip && ip%s.src == %s && outport == %s",
> is_v6 ? "6" : "4", nat->logical_ip,
> - od->l3dgw_port->json_key);
> - if (!distributed && od->l3redirect_port) {
> + od->l3dgw_ports[0]->json_key);
> + if (!distributed && od->n_l3dgw_ports) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the gateway chassis. */
> priority += 128;
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> }
> ds_clear(actions);
>
> @@ -12101,11 +12159,11 @@ build_lrouter_ingress_flow(struct hmap *lflows, struct ovn_datapath *od,
> struct ds *actions, struct eth_addr mac,
> bool distributed, bool is_v6)
> {
> - if (od->l3dgw_port && !strcmp(nat->type, "snat")) {
> + if (od->n_l3dgw_ports && !strcmp(nat->type, "snat")) {
> ds_clear(match);
> ds_put_format(
> match, "inport == %s && %s == %s",
> - od->l3dgw_port->json_key,
> + od->l3dgw_ports[0]->json_key,
> is_v6 ? "ip6.src" : "ip4.src", nat->external_ip);
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_IP_INPUT,
> 120, ds_cstr(match), "next;",
> @@ -12123,16 +12181,16 @@ build_lrouter_ingress_flow(struct hmap *lflows, struct ovn_datapath *od,
> */
> ds_clear(actions);
>
> - build_check_pkt_len_action_string(od->l3dgw_port, actions);
> + build_check_pkt_len_action_string(od->l3dgw_ports[0], actions);
> ds_put_format(actions, REG_INPORT_ETH_ADDR " = %s; next;",
> - od->l3dgw_port->lrp_networks.ea_s);
> + od->l3dgw_ports[0]->lrp_networks.ea_s);
>
> ds_clear(match);
> ds_put_format(match,
> "eth.dst == "ETH_ADDR_FMT" && inport == %s"
> " && is_chassis_resident(\"%s\")",
> ETH_ADDR_ARGS(mac),
> - od->l3dgw_port->json_key,
> + od->l3dgw_ports[0]->json_key,
> nat->logical_port);
> ovn_lflow_add_with_hint(lflows, od, S_ROUTER_IN_ADMISSION, 50,
> ds_cstr(match), ds_cstr(actions),
> @@ -12214,7 +12272,7 @@ lrouter_check_nat_entry(struct ovn_datapath *od, const struct nbrec_nat *nat,
> /* For distributed router NAT, determine whether this NAT rule
> * satisfies the conditions for distributed NAT processing. */
> *distributed = false;
> - if (od->l3dgw_port && !strcmp(nat->type, "dnat_and_snat") &&
> + if (od->n_l3dgw_ports && !strcmp(nat->type, "dnat_and_snat") &&
> nat->logical_port && nat->external_mac) {
> if (eth_addr_from_string(nat->external_mac, mac)) {
> *distributed = true;
> @@ -12259,7 +12317,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> * not committed, it would produce ongoing datapath flows with the ct.new
> * flag set. Some NICs are unable to offload these flows.
> */
> - if ((od->is_gw_router || od->l3dgw_port) &&
> + if ((od->is_gw_router || od->n_l3dgw_ports) &&
> (od->nbr->n_nat || od->nbr->n_load_balancer)) {
> ovn_lflow_add(lflows, od, S_ROUTER_OUT_UNDNAT, 50,
> "ip", "flags.loopback = 1; ct_dnat;");
> @@ -12275,7 +12333,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> /* NAT rules are only valid on Gateway routers and routers with
> * l3dgw_port (router has a port with gateway chassis
> * specified). */
> - if (!od->is_gw_router && !od->l3dgw_port) {
> + if (!od->is_gw_router && !od->n_l3dgw_ports) {
> return;
> }
>
> @@ -12316,14 +12374,14 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> ds_clear(match);
> ds_put_format(
> match, "outport == %s && %s == %s",
> - od->l3dgw_port->json_key,
> + od->l3dgw_ports[0]->json_key,
> is_v6 ? REG_NEXT_HOP_IPV6 : REG_NEXT_HOP_IPV4,
> nat->external_ip);
> ds_clear(actions);
> ds_put_format(
> actions, "eth.dst = %s; next;",
> distributed ? nat->external_mac :
> - od->l3dgw_port->lrp_networks.ea_s);
> + od->l3dgw_ports[0]->lrp_networks.ea_s);
> ovn_lflow_add_with_hint(lflows, od,
> S_ROUTER_IN_ARP_RESOLVE,
> 100, ds_cstr(match),
> @@ -12359,7 +12417,7 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> ds_put_format(match,
> "ip%s.src == %s && outport == %s",
> is_v6 ? "6" : "4", nat->logical_ip,
> - od->l3dgw_port->json_key);
> + od->l3dgw_ports[0]->json_key);
> /* Add a rule to drop traffic from a distributed NAT if
> * the virtual port has not claimed yet becaused otherwise
> * the traffic will be centralized misconfiguring the TOR switch.
> @@ -12386,16 +12444,16 @@ build_lrouter_nat_defrag_and_lb(struct ovn_datapath *od, struct hmap *lflows,
> * gateway port have ip.dst matching a NAT external IP, then
> * loop a clone of the packet back to the beginning of the
> * ingress pipeline with inport = outport. */
> - if (od->l3dgw_port) {
> + if (od->n_l3dgw_ports) {
> /* Distributed router. */
> ds_clear(match);
> ds_put_format(match, "ip%s.dst == %s && outport == %s",
> is_v6 ? "6" : "4",
> nat->external_ip,
> - od->l3dgw_port->json_key);
> + od->l3dgw_ports[0]->json_key);
> if (!distributed) {
> ds_put_format(match, " && is_chassis_resident(%s)",
> - od->l3redirect_port->json_key);
> + od->l3dgw_ports[0]->cr_port->json_key);
> } else {
> ds_put_format(match, " && is_chassis_resident(\"%s\")",
> nat->logical_port);
> diff --git a/northd/ovn_northd.dl b/northd/ovn_northd.dl
> index d7141294e..9c5576d16 100644
> --- a/northd/ovn_northd.dl
> +++ b/northd/ovn_northd.dl
> @@ -185,7 +185,7 @@ OutProxy_Port_Binding(._uuid = lsp._uuid,
> },
> Some{var router_port} = lsp.options.get("router-port"),
> var opt_chassis = peer.and_then(|p| p.router.options.get("chassis")),
> - var l3dgw_port = peer.and_then(|p| p.router.l3dgw_port),
> + var l3dgw_port = peer.and_then(|p| p.router.l3dgw_ports.nth(0)),
> (var __type, var options) = {
> var options = ["peer" -> router_port];
> match (opt_chassis) {
> @@ -241,7 +241,7 @@ OutProxy_Port_Binding(._uuid = lsp._uuid,
> Some{rport} -> match (
> (rport.lrp.options.get_bool_def("reside-on-redirect-chassis", false)
> and l3dgw_port.is_some()) or
> - Some{rport.lrp} == l3dgw_port or
> + rport.is_redirect or
> (rport.router.options.contains_key("chassis") and
> not sw.localnet_ports.is_empty())) {
> false -> set_empty(),
> @@ -335,7 +335,7 @@ function get_router_load_balancer_ips(router: Intern<Router>,
> function get_nat_addresses(rport: Intern<RouterPort>, routable_only: bool): Set<string> =
> {
> var addresses = set_empty();
> - var has_redirect = rport.router.l3dgw_port.is_some();
> + var has_redirect = not rport.router.l3dgw_ports.is_empty();
> match (eth_addr_from_string(rport.lrp.mac)) {
> None -> addresses,
> Some{mac} -> {
> @@ -402,7 +402,10 @@ function get_nat_addresses(rport: Intern<RouterPort>, routable_only: bool): Set<
> /* Gratuitous ARP for centralized NAT rules on distributed gateway
> * ports should be restricted to the gateway chassis. */
> if (has_redirect) {
> - c_addresses = c_addresses ++ " is_chassis_resident(${rport.router.redirect_port_name})"
> + c_addresses = c_addresses ++ match (rport.router.l3dgw_ports.nth(0)) {
> + None -> "",
> + Some {var gw_port} -> " is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})"
> + }
> } else ();
>
> addresses.insert(c_addresses)
> @@ -417,8 +420,10 @@ function get_garp_nat_addresses(rport: Intern<RouterPort>): string = {
> for (ipv4_addr in rport.networks.ipv4_addrs) {
> garp_info.push("${ipv4_addr.addr}")
> };
> - if (rport.router.redirect_port_name != "") {
> - garp_info.push("is_chassis_resident(${rport.router.redirect_port_name})")
> + match (rport.router.l3dgw_ports.nth(0)) {
> + None -> (),
> + Some {var gw_port} -> garp_info.push(
> + "is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})")
> };
> garp_info.join(" ")
> }
> @@ -455,7 +460,7 @@ OutProxy_Port_Binding(// lrp._uuid is already in use; generate a new UUID by
> .nat_addresses = set_empty(),
> .external_ids = lrp.external_ids) :-
> DistributedGatewayPort(lrp, lr_uuid),
> - LogicalRouterHAChassisGroup(lr_uuid, hacg_uuid),
> + DistributedGatewayPortHAChassisGroup(lrp, hacg_uuid),
> var redirect_type = match (lrp.options.get("redirect-type")) {
> Some{var value} -> ["redirect-type" -> value],
> _ -> map_empty()
> @@ -511,7 +516,8 @@ sb::Out_Port_Binding(._uuid = pbinding._uuid,
> * chassis. RefChassisSet has a row for every logical router. */
> relation RefChassis(lr_uuid: uuid, chassis_uuid: uuid)
> RefChassis(lr_uuid, chassis_uuid) :-
> - LogicalRouterHAChassisGroup(lr_uuid, _),
> + DistributedGatewayPortHAChassisGroup(lrp, _),
> + DistributedGatewayPort(lrp, lr_uuid),
> ConnectedLogicalRouter[(lr_uuid, set_uuid)],
> ConnectedLogicalRouter[(lr2_uuid, set_uuid)],
> FirstHopLogicalRouter(lr2_uuid, ls_uuid),
> @@ -538,7 +544,8 @@ RefChassisSet(lr_uuid, set_empty()) :-
> relation HAChassisGroupRefChassisSet(hacg_uuid: uuid,
> chassis_uuids: Set<uuid>)
> HAChassisGroupRefChassisSet(hacg_uuid, chassis_uuids) :-
> - LogicalRouterHAChassisGroup(lr_uuid, hacg_uuid),
> + DistributedGatewayPortHAChassisGroup(lrp, hacg_uuid),
> + DistributedGatewayPort(lrp, lr_uuid),
> RefChassisSet(lr_uuid, chassis_uuids),
> var chassis_uuids = chassis_uuids.group_by(hacg_uuid).union().
>
> @@ -4451,7 +4458,7 @@ for (&SwitchPort(.lsp = lsp,
> .peer = Some{&RouterPort{.lrp = lrp,
> .is_redirect = is_redirect,
> .router = &Router{._uuid = lr_uuid,
> - .redirect_port_name = redirect_port_name}}})
> + .l3dgw_ports = l3dgw_ports}}})
> if (lsp.addresses.contains("router") and lsp.__type != "external"))
> {
> Some{var mac} = scan_eth_addr(lrp.mac) in {
> @@ -4471,6 +4478,14 @@ for (&SwitchPort(.lsp = lsp,
> */
> lrp.options.get_bool_def("reside-on-redirect-chassis", false)) in
> var __match = if (add_chassis_resident_check) {
> + var redirect_port_name = if (is_redirect) {
> + json_string_escape(chassis_redirect_name(lrp.name))
> + } else {
> + match (l3dgw_ports.nth(0)) {
> + Some {var gw_port} -> json_string_escape(chassis_redirect_name(gw_port.name)),
> + None -> ""
> + }
> + };
> /* The destination lookup flow for the router's
> * distributed gateway port MAC address should only be
> * programmed on the "redirect-chassis". */
> @@ -4876,13 +4891,8 @@ var rLNIR = rEGBIT_LOOKUP_NEIGHBOR_IP_RESULT() in
>
> /* Check if we need to learn mac-binding from ARP requests. */
> for (RouterPortNetworksIPv4Addr(rp@&RouterPort{.router = router}, addr)) {
> - var is_l3dgw_port = match (router.l3dgw_port) {
> - Some{l3dgw_lrp} -> l3dgw_lrp._uuid == rp.lrp._uuid,
> - None -> false
> - } in
> - var has_redirect_port = router.redirect_port_name != "" in
> - var chassis_residence = match (is_l3dgw_port and has_redirect_port) {
> - true -> " && is_chassis_resident(${router.redirect_port_name})",
> + var chassis_residence = match (rp.is_redirect) {
> + true -> " && is_chassis_resident(${json_string_escape(chassis_redirect_name(rp.lrp.name))})",
> false -> ""
> } in
> var rLNR = rEGBIT_LOOKUP_NEIGHBOR_RESULT() in
> @@ -5042,7 +5052,7 @@ relation AddChassisResidentCheck_(lrp: uuid, add_check: bool)
> AddChassisResidentCheck_(lrp._uuid, res) :-
> &SwitchPort(.peer = Some{&RouterPort{.lrp = lrp, .router = router, .is_redirect = is_redirect}},
> .sw = sw),
> - router.l3dgw_port.is_some(),
> + not router.l3dgw_ports.is_empty(),
> not sw.localnet_ports.is_empty(),
> var res = if (is_redirect) {
> /* Traffic with eth.src = l3dgw_port->lrp_networks.ea
> @@ -5147,7 +5157,8 @@ LogicalRouterArpNdFlow(router, nat, None, rEG_INPORT_ETH_ADDR(), None, false, 90
> * different ETH address.
> */
> LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port) :-
> - router in &Router(._uuid = lr_uuid, .l3dgw_port = Some{l3dgw_port}),
> + router in &Router(._uuid = lr_uuid, .l3dgw_ports = l3dgw_ports),
> + Some {var l3dgw_port} = l3dgw_ports.nth(0),
> LogicalRouterNAT(lr_uuid, nat),
> /* Skip SNAT entries for now, we handle unique SNAT IPs separately
> * below.
> @@ -5155,7 +5166,8 @@ LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port) :-
> nat.nat.__type != "snat".
> /* Now handle SNAT entries too, one per unique SNAT IP. */
> LogicalRouterPortNatArpNdFlow(router, nat, l3dgw_port) :-
> - router in &Router(.l3dgw_port = Some{l3dgw_port}, .snat_ips = snat_ips),
> + router in &Router(.l3dgw_ports = l3dgw_ports, .snat_ips = snat_ips),
> + Some {var l3dgw_port} = l3dgw_ports.nth(0),
> var snat_ip = FlatMap(snat_ips),
> (var ip, var nats) = snat_ip,
> Some{var nat} = nats.nth(0).
> @@ -5185,9 +5197,9 @@ LogicalRouterArpNdFlow(router, nat, Some{lrp}, mac, None, true, 91) :-
> * upstream MAC learning points to the gateway chassis.
> * Also need to avoid generation of multiple ARP responses
> * from different chassis. */
> - match (router.redirect_port_name) {
> - "" -> "",
> - s -> "is_chassis_resident(${s})"
> + match (router.l3dgw_ports.nth(0)) {
> + None -> "",
> + Some {var gw_port} -> "is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})"
> }
> )
> }.
> @@ -5332,7 +5344,15 @@ for (RouterPortNetworksIPv4Addr(.port = &RouterPort{.lrp = lrp,
> var __match =
> "arp.spa == ${addr.match_network()}" ++
> if (add_chassis_resident_check) {
> - " && is_chassis_resident(${router.redirect_port_name})"
> + var redirect_port_name = if (is_redirect) {
> + json_string_escape(chassis_redirect_name(lrp.name))
> + } else {
> + match (router.l3dgw_ports.nth(0)) {
> + None -> "",
> + Some {var gw_port} -> json_string_escape(chassis_redirect_name(gw_port.name))
> + }
> + };
> + " && is_chassis_resident(${redirect_port_name})"
> } else "" in
> LogicalRouterArpFlow(.lr = router,
> .lrp = Some{lrp},
> @@ -5351,7 +5371,7 @@ for (&RouterPort(.lrp = lrp,
> .networks = networks,
> .is_redirect = is_redirect))
> var residence_check = match (is_redirect) {
> - true -> Some{"is_chassis_resident(${router.redirect_port_name})"},
> + true -> Some{"is_chassis_resident(${json_string_escape(chassis_redirect_name(lrp.name))})"},
> false -> None
> } in {
> (var all_ips_v4, _) = get_router_load_balancer_ips(router, false) in {
> @@ -5421,7 +5441,7 @@ Flow(.logical_datapath = lr_uuid,
> for (RouterPortNetworksIPv4Addr(
> .port = &RouterPort{
> .router = &Router{._uuid = lr_uuid,
> - .l3dgw_port = None,
> + .l3dgw_ports = vec_empty(),
> .is_gateway = false,
> .copp = copp},
> .lrp = lrp},
> @@ -5557,7 +5577,7 @@ for (RouterPortNetworksIPv6Addr(.port = &RouterPort{.lrp = lrp,
> /* UDP/TCP/SCTP port unreachable */
> for (RouterPortNetworksIPv6Addr(
> .port = &RouterPort{.router = &Router{._uuid = lr_uuid,
> - .l3dgw_port = None,
> + .l3dgw_ports = vec_empty(),
> .is_gateway = false,
> .copp = copp},
> .lrp = lrp,
> @@ -5685,11 +5705,11 @@ for (r in &Router(._uuid = lr_uuid)) {
> }
>
> for (r in &Router(._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> + .l3dgw_ports = l3dgw_ports,
> .is_gateway = is_gateway,
> .nat = nat,
> .load_balancer = load_balancer)
> - if (l3dgw_port.is_some() or is_gateway) and (not is_empty(nat) or not is_empty(load_balancer))) {
> + if (l3dgw_ports.len() > 0 or is_gateway) and (not is_empty(nat) or not is_empty(load_balancer))) {
> /* If the router has load balancer or DNAT rules, re-circulate every packet
> * through the DNAT zone so that packets that need to be unDNATed in the
> * reverse direction get unDNATed.
> @@ -5772,7 +5792,7 @@ function lrouter_nat_add_ext_ip_match(
> },
> false -> {
> /* S_ROUTER_OUT_SNAT uses priority (mask + 1 + 128 + 1) */
> - var is_gw_router = router.l3dgw_port == None;
> + var is_gw_router = router.l3dgw_ports.is_empty();
> var mask_1bits = mask.cidr_bits().unwrap_or(8'd0) as integer;
> mask_1bits + 2 + { if (not is_gw_router) 128 else 0 }
> }
> @@ -5877,10 +5897,9 @@ VirtualLogicalPort(Some{logical_port}) :-
> * l3dgw_port (router has a port with "redirect-chassis"
> * specified). */
> for (r in &Router(._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name = redirect_port_name,
> + .l3dgw_ports = l3dgw_ports,
> .is_gateway = is_gateway)
> - if l3dgw_port.is_some() or is_gateway)
> + if not l3dgw_ports.is_empty() or is_gateway)
> {
> for (LogicalRouterNAT(.lr = lr_uuid, .nat = nat)) {
> var ipX = nat.external_ip.ipX() in
> @@ -5898,7 +5917,7 @@ for (r in &Router(._uuid = lr_uuid,
> } in
> /* For distributed router NAT, determine whether this NAT rule
> * satisfies the conditions for distributed NAT processing. */
> - var mac = match ((l3dgw_port.is_some() and nat.nat.__type == "dnat_and_snat",
> + var mac = match ((not l3dgw_ports.is_empty() and nat.nat.__type == "dnat_and_snat",
> nat.nat.logical_port, nat.external_mac)) {
> (true, Some{_}, Some{mac}) -> Some{mac},
> _ -> None
> @@ -5916,7 +5935,7 @@ for (r in &Router(._uuid = lr_uuid,
> * not know about the possibility of eventual additional SNAT in
> * egress pipeline. */
> if (nat.nat.__type == "snat" or nat.nat.__type == "dnat_and_snat") {
> - if (l3dgw_port == None) {
> + if (l3dgw_ports.is_empty()) {
> /* Gateway router. */
> var actions = if (stateless) {
> "${ipX}.dst=${nat.nat.logical_ip}; next;"
> @@ -5930,7 +5949,7 @@ for (r in &Router(._uuid = lr_uuid,
> .actions = actions,
> .external_ids = stage_hint(nat.nat._uuid))
> };
> - Some{var gwport} = l3dgw_port in {
> + Some {var gwport} = l3dgw_ports.nth(0) in {
> /* Distributed router. */
>
> /* Traffic received on l3dgw_port is subject to NAT. */
> @@ -5940,7 +5959,7 @@ for (r in &Router(._uuid = lr_uuid,
> if (mac == None) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " && is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> var actions = if (stateless) {
> "${ipX}.dst=${nat.nat.logical_ip}; next;"
> @@ -5966,7 +5985,7 @@ for (r in &Router(._uuid = lr_uuid,
> ""
> } in
> if (nat.nat.__type == "dnat" or nat.nat.__type == "dnat_and_snat") {
> - None = l3dgw_port in
> + l3dgw_ports.is_empty() in
> var __match = "ip && ${ipX}.dst == ${nat.nat.external_ip}" in
> (var ext_ip_match, var ext_flow) = lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, true, mask) in
> @@ -5998,14 +6017,14 @@ for (r in &Router(._uuid = lr_uuid,
> .external_ids = stage_hint(nat.nat._uuid))
> };
>
> - Some{var gwport} = l3dgw_port in
> + Some {var gwport} = l3dgw_ports.nth(0) in
> var __match =
> "ip && ${ipX}.dst == ${nat.nat.external_ip}"
> " && inport == ${json_string_escape(gwport.name)}" ++
> if (mac == None) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " && is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> (var ext_ip_match, var ext_flow) = lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, true, mask) in
> @@ -6029,7 +6048,7 @@ for (r in &Router(._uuid = lr_uuid,
> };
>
> /* ARP resolve for NAT IPs. */
> - Some{var gwport} = l3dgw_port in {
> + Some {var gwport} = l3dgw_ports.nth(0) in {
> var gwport_name = json_string_escape(gwport.name) in {
> if (nat.nat.__type == "snat") {
> var __match = "inport == ${gwport_name} && "
> @@ -6066,14 +6085,14 @@ for (r in &Router(._uuid = lr_uuid,
> * Note that this only applies for NAT on a distributed router.
> */
> if ((nat.nat.__type == "dnat" or nat.nat.__type == "dnat_and_snat")) {
> - Some{var gwport} = l3dgw_port in
> + Some {var gwport} = l3dgw_ports.nth(0) in
> var __match =
> "ip && ${ipX}.src == ${nat.nat.logical_ip}"
> " && outport == ${json_string_escape(gwport.name)}" ++
> if (mac == None) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " && is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> var actions =
> match (mac) {
> @@ -6103,7 +6122,7 @@ for (r in &Router(._uuid = lr_uuid,
> ""
> } in
> if (nat.nat.__type == "snat" or nat.nat.__type == "dnat_and_snat") {
> - None = l3dgw_port in
> + l3dgw_ports.is_empty() in
> var __match = "ip && ${ipX}.src == ${nat.nat.logical_ip}" in
> (var ext_ip_match, var ext_flow) = lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, false, mask) in
> @@ -6128,14 +6147,14 @@ for (r in &Router(._uuid = lr_uuid,
> .external_ids = stage_hint(nat.nat._uuid))
> };
>
> - Some{var gwport} = l3dgw_port in
> + Some {var gwport} = l3dgw_ports.nth(0) in
> var __match =
> "ip && ${ipX}.src == ${nat.nat.logical_ip}"
> " && outport == ${json_string_escape(gwport.name)}" ++
> if (mac == None) {
> /* Flows for NAT rules that are centralized are only
> * programmed on the "redirect-chassis". */
> - " && is_chassis_resident(${redirect_port_name})"
> + " && is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})"
> } else { "" } in
> (var ext_ip_match, var ext_flow) = lrouter_nat_add_ext_ip_match(
> r, nat, __match, ipX, false, mask) in
> @@ -6173,7 +6192,7 @@ for (r in &Router(._uuid = lr_uuid,
> * on the l3dgw_port instance where nat->logical_port is
> * resident. */
> Some{var mac_addr} = mac in
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> Some{var logical_port} = nat.nat.logical_port in
> var __match =
> "eth.dst == ${mac_addr} && inport == ${json_string_escape(gwport.name)}"
> @@ -6199,7 +6218,7 @@ for (r in &Router(._uuid = lr_uuid,
> * stage is sent out with proper IP/MAC src addresses
> */
> Some{var mac_addr} = mac in
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> Some{var logical_port} = nat.nat.logical_port in
> Some{var external_mac} = nat.nat.external_mac in
> var __match =
> @@ -6218,7 +6237,7 @@ for (r in &Router(._uuid = lr_uuid,
> .external_ids = stage_hint(nat.nat._uuid));
>
> for (VirtualLogicalPort(nat.nat.logical_port)) {
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> Flow(.logical_datapath = lr_uuid,
> .stage = s_ROUTER_IN_GW_REDIRECT(),
> .priority = 80,
> @@ -6233,14 +6252,14 @@ for (r in &Router(._uuid = lr_uuid,
> * gateway port have ip.dst matching a NAT external IP, then
> * loop a clone of the packet back to the beginning of the
> * ingress pipeline with inport = outport. */
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> /* Distributed router. */
> Some{var port} = match (mac) {
> Some{_} -> match (nat.nat.logical_port) {
> Some{name} -> Some{json_string_escape(name)},
> None -> None: Option<string>
> },
> - None -> Some{redirect_port_name}
> + None -> Some{json_string_escape(chassis_redirect_name(gwport.name))}
> } in
> var __match = "${ipX}.dst == ${nat.nat.external_ip} && outport == ${json_string_escape(gwport.name)} && is_chassis_resident(${port})" in
> var regs = {
> @@ -6268,7 +6287,7 @@ for (r in &Router(._uuid = lr_uuid,
> };
>
> /* Handle force SNAT options set in the gateway router. */
> - if (l3dgw_port == None) {
> + if (l3dgw_ports.is_empty()) {
> var dnat_force_snat_ips = get_force_snat_ip(r.options, "dnat") in
> if (not dnat_force_snat_ips.is_empty())
> LogicalRouterForceSnatFlows(.logical_router = lr_uuid,
> @@ -6296,14 +6315,13 @@ function nats_contain_vip(nats: Vec<NAT>, vip: v46_ip): bool {
> * Gateway routers or router with gateway port. */
> for (RouterLBVIP(
> .router = r@&Router{._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name = redirect_port_name,
> + .l3dgw_ports = l3dgw_ports,
> .is_gateway = is_gateway,
> .nats = nats},
> .lb = lb,
> .vip = vip,
> .backends = backends)
> - if l3dgw_port.is_some() or is_gateway)
> + if not l3dgw_ports.is_empty() or is_gateway)
> {
> if (backends == "" and not lb.options.get_bool_def("reject", false)) {
> for (LoadBalancerEmptyEvents(lb)) {
> @@ -6372,8 +6390,8 @@ for (RouterLBVIP(
> (110, "")
> } in
> var __match = match1 ++ match2 ++
> - match ((l3dgw_port, backends != "" or lb.options.get_bool_def("reject", false))) {
> - (Some{gwport}, true) -> " && is_chassis_resident(${redirect_port_name})",
> + match ((l3dgw_ports.nth(0), backends != "" or lb.options.get_bool_def("reject", false))) {
> + (Some{gw_port}, true) -> " && is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})",
> _ -> ""
> } in
> var snat_for_lb = snat_for_lb(r.options, lb) in
> @@ -6385,8 +6403,8 @@ for (RouterLBVIP(
> } else {
> ""
> } ++
> - match ((l3dgw_port, backends != "" or lb.options.get_bool_def("reject", false))) {
> - (Some{gwport}, true) -> " && is_chassis_resident(${redirect_port_name})",
> + match ((l3dgw_ports.nth(0), backends != "" or lb.options.get_bool_def("reject", false))) {
> + (Some {var gw_port}, true) -> " && is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})",
> _ -> ""
> } in
> var actions =
> @@ -6425,7 +6443,7 @@ for (RouterLBVIP(
> .external_ids = stage_hint(lb._uuid))
> };
>
> - Some{var gwport} = l3dgw_port in
> + Some{var gwport} = l3dgw_ports.nth(0) in
> /* Add logical flows to UNDNAT the load balanced reverse traffic in
> * the router egress pipleine stage - S_ROUTER_OUT_UNDNAT if the logical
> * router has a gateway router port associated.
> @@ -6450,7 +6468,7 @@ for (RouterLBVIP(
> var undnat_match =
> "${ip_address.ipX()} && (" ++ conds.join(" || ") ++
> ") && outport == ${json_string_escape(gwport.name)} && "
> - "is_chassis_resident(${redirect_port_name})" in
> + "is_chassis_resident(${json_string_escape(chassis_redirect_name(gwport.name))})" in
> var action =
> match (snat_for_lb) {
> SkipSNAT -> "flags.skip_snat_for_lb = 1; ct_dnat;",
> @@ -6481,14 +6499,14 @@ MeteredFlow(.logical_datapath = r._uuid,
> .controller_meter = meter,
> .external_ids = stage_hint(lb._uuid)) :-
> r in &Router(),
> - r.l3dgw_port.is_some() or r.is_gateway,
> + r.l3dgw_ports.len() > 0 or r.is_gateway,
> LBVIPWithStatus[lbvip@&LBVIPWithStatus{.lb = lb}],
> r.load_balancer.contains(lb._uuid),
> var __match
> = "ct.new && " ++
> get_match_for_lb_key(lbvip.vip_addr, lbvip.vip_port, lb.protocol, true, true) ++
> - match (r.l3dgw_port) {
> - Some{gwport} -> " && is_chassis_resident(${r.redirect_port_name})",
> + match (r.l3dgw_ports.nth(0)) {
> + Some{gw_port} -> " && is_chassis_resident(${json_string_escape(chassis_redirect_name(gw_port.name))})",
> _ -> ""
> },
> var priority = if (lbvip.vip_port != 0) 120 else 110,
> @@ -7294,11 +7312,11 @@ Flow(.logical_datapath = router._uuid,
> .stage = s_ROUTER_IN_ARP_RESOLVE(),
> .priority = 50,
> .__match = "outport == ${rp.json_name} && "
> - "!is_chassis_resident(${router.redirect_port_name})",
> + "!is_chassis_resident(${json_string_escape(chassis_redirect_name(l3dgw_port.name))})",
> .actions = "eth.dst = ${rp.networks.ea}; next;",
> .external_ids = stage_hint(lrp._uuid)) :-
> rp in &RouterPort(.lrp = lrp, .router = router),
> - router.redirect_port_name != "",
> + Some{var l3dgw_port} = router.l3dgw_ports.nth(0),
> Some{"bridged"} = lrp.options.get("redirect-type").
>
>
> @@ -7672,21 +7690,20 @@ MeteredFlow(.logical_datapath = lr_uuid,
> * of the traffic to the l3redirect_port which represents
> * the central instance of the l3dgw_port.
> */
> -for (&Router(._uuid = lr_uuid,
> - .l3dgw_port = l3dgw_port,
> - .redirect_port_name = redirect_port_name))
> +for (&Router(._uuid = lr_uuid))
> {
> /* For traffic with outport == l3dgw_port, if the
> * packet did not match any higher priority redirect
> * rule, then the traffic is redirected to the central
> * instance of the l3dgw_port. */
> - Some{var gwport} = l3dgw_port in
> - Flow(.logical_datapath = lr_uuid,
> - .stage = s_ROUTER_IN_GW_REDIRECT(),
> - .priority = 50,
> - .__match = "outport == ${json_string_escape(gwport.name)}",
> - .actions = "outport = ${redirect_port_name}; next;",
> - .external_ids = stage_hint(gwport._uuid));
> + for (DistributedGatewayPort(lrp, lr_uuid)) {
> + Flow(.logical_datapath = lr_uuid,
> + .stage = s_ROUTER_IN_GW_REDIRECT(),
> + .priority = 50,
> + .__match = "outport == ${json_string_escape(lrp.name)}",
> + .actions = "outport = ${json_string_escape(chassis_redirect_name(lrp.name))}; next;",
> + .external_ids = stage_hint(lrp._uuid))
> + };
>
> /* Packets are allowed by default. */
> Flow(.logical_datapath = lr_uuid,
> diff --git a/ovn-architecture.7.xml b/ovn-architecture.7.xml
> index 0eef9b739..3598b5073 100644
> --- a/ovn-architecture.7.xml
> +++ b/ovn-architecture.7.xml
> @@ -731,6 +731,13 @@
> highest-priority gateway that is online.
> </p>
>
> + <p>
> + A logical router can have multiple distributed gateway ports, each
> + connecting different external networks. However, some features, such as NAT
> + and load balancers, are not supported yet for logical routers with more
> + than one distributed gateway port configured.
> + </p>
> +
> <h4>Physical VLAN MTU Issues</h4>
>
> <p>
> @@ -1968,8 +1975,9 @@
>
> <p>
> If the logical router doesn't have a distributed gateway port connecting
> - to the localnet logical switch which provides external connectivity,
> - then this option is ignored by <code>OVN</code>.
> + to the localnet logical switch which provides external connectivity, or
> + if it has more than one distributed gateway ports, then this option is
> + ignored by <code>OVN</code>.
> </p>
>
> <p>
> @@ -2086,6 +2094,13 @@
> a tunnel.
> </p>
>
> + <p>
> + If the logical router doesn't have a distributed gateway port connecting
> + to the localnet logical switch which provides external connectivity, or
> + if it has more than one distributed gateway ports, then this option is
> + ignored by <code>OVN</code>.
> + </p>
> +
> <p>
> Following happens for bridged redirection:
> </p>
> diff --git a/ovn-nb.xml b/ovn-nb.xml
> index c1176e81f..ec51b5608 100644
> --- a/ovn-nb.xml
> +++ b/ovn-nb.xml
> @@ -2032,13 +2032,14 @@
>
> <column name="nat">
> One or more NAT rules for the router. NAT rules only work on
> - Gateway routers, and on distributed routers with logical gateway ports.
> + Gateway routers, and on distributed routers with one and only one
> + distributed gateway port.
> </column>
>
> <column name="load_balancer">
> Load balance a virtual ip address to a set of logical port ip
> addresses. Load balancer rules only work on the Gateway routers or
> - routers with distributed gateway ports.
> + routers with one and only one distributed gateway port.
> </column>
>
> <group title="Naming">
> @@ -2453,8 +2454,7 @@
> If either of these are set, this logical router port represents a
> distributed gateway port that connects this router to a
> logical switch with a <code>localnet</code> port or a
> - connection to another OVN deployment. There may be at most
> - one such logical router port on each logical router.
> + connection to another OVN deployment.
> </p>
>
> <p>
> @@ -2476,8 +2476,16 @@
> </p>
>
> <p>
> - When more than one gateway chassis is specified, OVN only uses
> - one at a time. OVN can rely on OVS BFD implementation to monitor
> + There can be more than one distributed gateway ports configured
> + on each logical router, each connecting to different L2 segments.
> + However, features such as NAT and load-balancer are not supported
> + on logical routers with more than one distributed gateway ports.
> + </p>
> +
> + <p>
> + For each distributed gateway port, it may have more than one gateway
> + chassises. When more than one gateway chassis is specified, OVN only
> + uses one at a time. OVN can rely on OVS BFD implementation to monitor
> gateway connectivity, preferring the highest-priority gateway
> that is online. Priorities are specified in the <code>priority</code>
> column of <ref table="Gateway_Chassis"/> or <ref table="HA_Chassis"/>.
> @@ -2563,8 +2571,8 @@
> </p>
>
> <p>
> - OVN honors this option only if the logical router has a distributed
> - gateway port and if the LRP's peer switch has a
> + OVN honors this option only if the logical router has one and only
> + one distributed gateway port and if the LRP's peer switch has a
> <code>localnet</code> port.
> </p>
> </column>
> @@ -2588,7 +2596,8 @@
> <p>
> Setting this option to <code>overlay</code> or leaving it unset has
> no effect. This option may usefully be set only on a distributed
> - gateway port. It is otherwise ignored.
> + gateway port when there is one and only one distributed gateway
> + port on the logical router. It is otherwise ignored.
> </p>
> </column>
> </group>
> diff --git a/tests/ovn-northd.at b/tests/ovn-northd.at
> index 2098b1c19..27c93a8b9 100644
> --- a/tests/ovn-northd.at
> +++ b/tests/ovn-northd.at
> @@ -4968,3 +4968,85 @@ AT_CHECK([grep -e "chk_pkt_len" -e "lr_in_larger_pkts" lr0flows | sort], [0], [d
>
> AT_CLEANUP
> ])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([ovn-northd -- lr multiple gw ports])
> +AT_KEYWORDS([multiple-l3dgw-ports])
> +ovn_start
> +
> +# Logical network:
> +# 1 Logical Router, 3 bridged Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# Validate basic LR logical flows.
> +
> +check ovn-sbctl chassis-add gw1 geneve 127.0.0.1
> +check ovn-sbctl chassis-add gw2 geneve 128.0.0.1
> +check ovn-sbctl chassis-add gw3 geneve 129.0.0.1
> +
> +check ovn-nbctl lr-add DR
> +check ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +check ovn-nbctl lrp-add DR DR-S2 03:ac:10:01:00:01 172.16.2.1/24
> +check ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 172.16.3.1/24
> +check ovn-nbctl lrp-add DR DR-ls 05:ac:10:01:00:01 20.0.0.1/24
> +
> +check ovn-nbctl ls-add S1
> +check ovn-nbctl lsp-add S1 S1-DR
> +check ovn-nbctl lsp-set-type S1-DR router
> +check ovn-nbctl lsp-set-addresses S1-DR router
> +check ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +
> +check ovn-nbctl ls-add S2
> +check ovn-nbctl lsp-add S2 S2-DR
> +check ovn-nbctl lsp-set-type S2-DR router
> +check ovn-nbctl lsp-set-addresses S2-DR router
> +check ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +
> +check ovn-nbctl ls-add S3
> +check ovn-nbctl lsp-add S3 S3-DR
> +check ovn-nbctl lsp-set-type S3-DR router
> +check ovn-nbctl lsp-set-addresses S3-DR router
> +check ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +
> +check ovn-nbctl ls-add ls
> +check ovn-nbctl lsp-add ls ls-DR
> +check ovn-nbctl lsp-set-type ls-DR router
> +check ovn-nbctl lsp-set-addresses ls-DR router
> +check ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +check ovn-nbctl lrp-set-gateway-chassis DR-S1 gw1
> +check ovn-nbctl lrp-set-gateway-chassis DR-S2 gw2
> +check ovn-nbctl lrp-set-gateway-chassis DR-S3 gw3
> +
> +check ovn-nbctl --wait=sb sync
> +
> +ovn-sbctl dump-flows DR > lrflows
> +AT_CAPTURE_FILE([lrflows])
> +
> +# Check the flows in lr_in_admission stage
> +AT_CHECK([grep lr_in_admission lrflows | grep cr-DR | sort], [0], [dnl
> + table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 02:ac:10:01:00:01 && inport == "DR-S1" && is_chassis_resident("cr-DR-S1")), action=(xreg0[[0..47]] = 02:ac:10:01:00:01; next;)
> + table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 03:ac:10:01:00:01 && inport == "DR-S2" && is_chassis_resident("cr-DR-S2")), action=(xreg0[[0..47]] = 03:ac:10:01:00:01; next;)
> + table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 04:ac:10:01:00:01 && inport == "DR-S3" && is_chassis_resident("cr-DR-S3")), action=(xreg0[[0..47]] = 04:ac:10:01:00:01; next;)
> +])
> +# Check the flows in lr_in_lookup_neighbor stage
> +AT_CHECK([grep lr_in_lookup_neighbor lrflows | grep cr-DR | sort], [0], [dnl
> + table=1 (lr_in_lookup_neighbor), priority=100 , match=(inport == "DR-S1" && arp.spa == 172.16.1.0/24 && arp.op == 1 && is_chassis_resident("cr-DR-S1")), action=(reg9[[2]] = lookup_arp(inport, arp.spa, arp.sha); next;)
> + table=1 (lr_in_lookup_neighbor), priority=100 , match=(inport == "DR-S2" && arp.spa == 172.16.2.0/24 && arp.op == 1 && is_chassis_resident("cr-DR-S2")), action=(reg9[[2]] = lookup_arp(inport, arp.spa, arp.sha); next;)
> + table=1 (lr_in_lookup_neighbor), priority=100 , match=(inport == "DR-S3" && arp.spa == 172.16.3.0/24 && arp.op == 1 && is_chassis_resident("cr-DR-S3")), action=(reg9[[2]] = lookup_arp(inport, arp.spa, arp.sha); next;)
> +])
> +# Check the flows in lr_in_gw_redirect stage
> +AT_CHECK([grep lr_in_gw_redirect lrflows | grep cr-DR | sort], [0], [dnl
> + table=17(lr_in_gw_redirect ), priority=50 , match=(outport == "DR-S1"), action=(outport = "cr-DR-S1"; next;)
> + table=17(lr_in_gw_redirect ), priority=50 , match=(outport == "DR-S2"), action=(outport = "cr-DR-S2"; next;)
> + table=17(lr_in_gw_redirect ), priority=50 , match=(outport == "DR-S3"), action=(outport = "cr-DR-S3"; next;)
> +])
> +
> +AT_CLEANUP
> +])
> diff --git a/tests/ovn.at b/tests/ovn.at
> index 7ae136ad9..b571bbb49 100644
> --- a/tests/ovn.at
> +++ b/tests/ovn.at
> @@ -27529,3 +27529,310 @@ AT_CHECK([ovs-ofctl dump-flows br-int table=44 | grep 10.0.0.144], [0], [ignore]
> OVN_CLEANUP([hv1])
> AT_CLEANUP
> ])
> +
> +OVN_FOR_EACH_NORTHD([
> +AT_SETUP([ovn -- lr multiple gw ports])
> +AT_KEYWORDS([multiple-l3dgw-ports])
> +ovn_start
> +
> +# Logical network:
> +# 1 LR, 3 Logical Switches,
> +# 1 gateway chassis attached to each corresponding LRP.
> +#
> +# | S1 (gw1)
> +# |
> +# ls ---- DR -- S3 (gw3)
> +# (20.0.0.0/24) |
> +# | S2 (gw2)
> +#
> +# S1 - VLAN 1000
> +# S2 - VLAN 2000
> +# S3 - VLAN 3000
> +#
> +# 5 chassis(s), HV1----HV5
> +#
> +# HV1 - VIF11
> +# HV2 - Gateway chassis gw1
> +# HV3 - Gateway chassis gw2
> +# HV4 - Gateway chassis gw3
> +# HV5 - North endpoint
> +
> +ovn-nbctl lr-add DR
> +ovn-nbctl lrp-add DR DR-S1 02:ac:10:01:00:01 172.16.1.1/24
> +ovn-nbctl lrp-add DR DR-S2 08:ac:10:01:00:01 10.0.0.1/24
> +ovn-nbctl lrp-add DR DR-S3 04:ac:10:01:00:01 192.168.0.1/24
> +ovn-nbctl lrp-add DR DR-ls 06:ac:10:01:00:01 20.0.0.1/24
> +
> +ovn-nbctl ls-add S1
> +ovn-nbctl lsp-add S1 S1-DR
> +ovn-nbctl lsp-set-type S1-DR router
> +ovn-nbctl lsp-set-addresses S1-DR router
> +ovn-nbctl --wait=sb lsp-set-options S1-DR router-port=DR-S1
> +ovn-nbctl lsp-add S1 ln1 "" 1000
> +ovn-nbctl lsp-set-addresses ln1 unknown
> +ovn-nbctl lsp-set-type ln1 localnet
> +ovn-nbctl lsp-set-options ln1 network_name=phys
> +
> +ovn-nbctl ls-add S2
> +ovn-nbctl lsp-add S2 S2-DR
> +ovn-nbctl lsp-set-type S2-DR router
> +ovn-nbctl lsp-set-addresses S2-DR router
> +ovn-nbctl --wait=sb lsp-set-options S2-DR router-port=DR-S2
> +ovn-nbctl lsp-add S2 ln2 "" 2000
> +ovn-nbctl lsp-set-addresses ln2 unknown
> +ovn-nbctl lsp-set-type ln2 localnet
> +ovn-nbctl lsp-set-options ln2 network_name=phys
> +
> +ovn-nbctl ls-add S3
> +ovn-nbctl lsp-add S3 S3-DR
> +ovn-nbctl lsp-set-type S3-DR router
> +ovn-nbctl lsp-set-addresses S3-DR router
> +ovn-nbctl --wait=sb lsp-set-options S3-DR router-port=DR-S3
> +ovn-nbctl lsp-add S3 ln3 "" 3000
> +ovn-nbctl lsp-set-addresses ln3 unknown
> +ovn-nbctl lsp-set-type ln3 localnet
> +ovn-nbctl lsp-set-options ln3 network_name=phys
> +
> +ovn-nbctl ls-add ls
> +ovn-nbctl lsp-add ls ls-DR
> +ovn-nbctl lsp-set-type ls-DR router
> +ovn-nbctl lsp-set-addresses ls-DR router
> +ovn-nbctl --wait=sb lsp-set-options ls-DR router-port=DR-ls
> +
> +# Add the lsp lp11 to ls. This will map to VIF11.
> +ovn-nbctl lsp-add ls lp11
> +ovn-nbctl lsp-set-addresses lp11 "f0:00:00:00:00:10 20.0.0.10"
> +ovn-nbctl lsp-set-port-security lp11 f0:00:00:00:00:10
> +
> +# Add the Northbound endpoint, lp-north1
> +ovn-nbctl ls-add ls-north1
> +ovn-nbctl lsp-add ls-north1 ln4 "" 1000
> +ovn-nbctl lsp-set-addresses ln4 unknown
> +ovn-nbctl lsp-set-type ln4 localnet
> +ovn-nbctl lsp-set-options ln4 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north1 lp-north1
> +ovn-nbctl lsp-set-addresses lp-north1 "f0:f0:00:00:00:11 172.16.1.10"
> +ovn-nbctl lsp-set-port-security lp-north1 f0:f0:00:00:00:11
> +
> +# Add the Northbound endpoint, lp-north2
> +ovn-nbctl ls-add ls-north2
> +ovn-nbctl lsp-add ls-north2 ln5 "" 2000
> +ovn-nbctl lsp-set-addresses ln5 unknown
> +ovn-nbctl lsp-set-type ln5 localnet
> +ovn-nbctl lsp-set-options ln5 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north2 lp-north2
> +ovn-nbctl lsp-set-addresses lp-north2 "f0:f0:00:00:00:22 10.0.0.10"
> +ovn-nbctl lsp-set-port-security lp-north2 f0:f0:00:00:00:22
> +
> +# Add the Northbound endpoint, lp-north3
> +ovn-nbctl ls-add ls-north3
> +ovn-nbctl lsp-add ls-north3 ln6 "" 3000
> +ovn-nbctl lsp-set-addresses ln6 unknown
> +ovn-nbctl lsp-set-type ln6 localnet
> +ovn-nbctl lsp-set-options ln6 network_name=phys
> +
> +ovn-nbctl lsp-add ls-north3 lp-north3
> +ovn-nbctl lsp-set-addresses lp-north3 "f0:f0:00:00:00:33 192.168.0.10"
> +ovn-nbctl lsp-set-port-security lp-north3 f0:f0:00:00:00:33
> +
> +# Add 5 chassis
> +net_add n1
> +for i in 1 2 3 4 5; do
> + sim_add hv$i
> + as hv$i
> + ovs-vsctl add-br br-phys
> + ovs-vsctl set open . external-ids:ovn-bridge-mappings=phys:br-phys
> + ovn_attach n1 br-phys 192.168.0.$i 24 $encap
> +done
> +
> +# Add a vif on HV1
> +as hv1 ovs-vsctl add-port br-int vif11 -- \
> + set Interface vif11 external-ids:iface-id=lp11 \
> + options:tx_pcap=hv1/vif11-tx.pcap \
> + options:rxq_pcap=hv1/vif11-rx.pcap \
> + ofport-request=11
> +OVS_WAIT_UNTIL([test x`ovn-nbctl lsp-get-up lp11` = xup])
> +
> +as hv5 ovs-vsctl add-port br-int vif-north1 -- \
> + set Interface vif-north1 external-ids:iface-id=lp-north1 \
> + options:tx_pcap=hv5/vif-north1-tx.pcap \
> + options:rxq_pcap=hv5/vif-north1-rx.pcap \
> + ofport-request=44
> +
> +as hv5 ovs-vsctl add-port br-int vif-north2 -- \
> + set Interface vif-north2 external-ids:iface-id=lp-north2 \
> + options:tx_pcap=hv5/vif-north2-tx.pcap \
> + options:rxq_pcap=hv5/vif-north2-rx.pcap \
> + ofport-request=45
> +
> +as hv5 ovs-vsctl add-port br-int vif-north3 -- \
> + set Interface vif-north3 external-ids:iface-id=lp-north3 \
> + options:tx_pcap=hv5/vif-north3-tx.pcap \
> + options:rxq_pcap=hv5/vif-north3-rx.pcap \
> + ofport-request=46
> +
> +ovn-nbctl lrp-set-gateway-chassis DR-S1 hv2
> +ovn-nbctl lrp-set-gateway-chassis DR-S2 hv3
> +ovn-nbctl lrp-set-gateway-chassis DR-S3 hv4
> +
> +ovn-nbctl --wait=sb sync
> +OVN_POPULATE_ARP
> +
> +vif_to_ls () {
> + case ${1} in dnl (
> + vif?[[11]]) echo ls ;; dnl (
> + vif-north1) echo ls-north1 ;; dnl (
> + vif-north2) echo ls-north2 ;; dnl (
> + vif-north3) echo ls-north3 ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +}
> +
> +vif_to_hv () {
> + case ${1} in dnl (
> + vif[[1]]?) echo hv1 ;; dnl (
> + vif-north1) echo hv5 ;; dnl (
> + vif-north2) echo hv5 ;; dnl (
> + vif-north3) echo hv5 ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +}
> +
> +vif_to_lrp () {
> + case ${1} in dnl (
> + vif?[[11]]) echo DR-ls ;; dnl (
> + *) AT_FAIL_IF([:]) ;;
> + esac
> +
> +}
> +
> +ip_to_hex() {
> + printf "%02x%02x%02x%02x" "${@}"
> +}
> +
> +# test_arp INPORT SHA SPA TPA
> +#
> +# Causes a packet to be received on INPORT. The packet is an ARP
> +# request with SHA, SPA, and TPA as specified.
> +test_arp() {
> + local inport=$1 sha=$2 spa=$3 tpa=$4
> + local request=ffffffffffff${sha}08060001080006040001${sha}${spa}ffffffffffff${tpa}
> + hv=`vif_to_hv $inport`
> + as $hv ovs-appctl netdev-dummy/receive $inport $request
> +}
> +
> +
> +test_ip() {
> + # This packet has bad checksums but logical L3 routing doesn't check.
> + local inport=${1} src_mac=${2} dst_mac=${3} src_ip=${4} dst_ip=${5} outport=${6}
> + local packet=${dst_mac}${src_mac}08004500001c0000000040110000${src_ip}${dst_ip}0035111100080000
> + shift; shift; shift; shift; shift
> + hv=`vif_to_hv $inport`
> + as $hv ovs-appctl netdev-dummy/receive $inport $packet
> + in_ls=`vif_to_ls $inport`
> + for outport; do
> + out_ls=`vif_to_ls $outport`
> + if test $in_ls = $out_ls; then
> + # Ports on the same logical switch receive exactly the same packet.
> + echo $packet
> + else
> + # Routing decrements TTL and updates source and dest MAC
> + # (and checksum).
> + # For North-South, packet will come via gateway chassis, i.e hv3
> + if test $inport = vif-north1; then
> + echo f0000000001006ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + if test $outport = vif-north1; then
> + echo f0f00000001102ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + if test $outport = vif-north2; then
> + echo f0f00000002208ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + if test $outport = vif-north3; then
> + echo f0f00000003304ac1001000108004500001c000000003f110100${src_ip}${dst_ip}0035111100080000 >> $outport.expected
> + fi
> + fi >> $outport.expected
> + done
> +}
> +
> +echo "------ OVN dump ------"
> +ovn-nbctl show
> +ovn-sbctl show
> +ovn-sbctl list port_binding
> +ovn-sbctl list mac_binding
> +ovn-sbctl list datapath_binding
> +
> +ovn-sbctl dump-flows DR
> +ovn-sbctl dump-flows S1
> +ovn-sbctl dump-flows ls
> +
> +echo "------ hv1 dump ------"
> +as hv1 ovs-vsctl show
> +as hv1 ovs-vsctl list Open_Vswitch
> +as hv1 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv2 dump ------"
> +as hv2 ovs-vsctl show
> +as hv2 ovs-vsctl list Open_Vswitch
> +as hv2 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv3 dump ------"
> +as hv3 ovs-vsctl show
> +as hv3 ovs-vsctl list Open_Vswitch
> +as hv3 ovs-ofctl dump-flows br-int
> +
> +echo "------ hv4 dump ------"
> +as hv4 ovs-vsctl show
> +as hv4 ovs-vsctl list Open_Vswitch
> +as hv5 ovs-ofctl dump-flows br-int
> +
> +# N-S with lp-north1
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 172 16 1 10`
> +tip=`ip_to_hex 172 16 1 50`
> +test_arp vif-north1 f0f000000011 $sip $tip
> +
> +echo "Send traffic North to South"
> +sip=`ip_to_hex 172 16 1 10`
> +dip=`ip_to_hex 20 0 0 10`
> +test_ip vif-north1 f0f000000011 02ac10010001 $sip $dip vif11
> +# Confirm that North to south traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv1/vif11-tx.pcap], [vif11.expected])
> +
> +echo "Send traffic South to North1"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 172 16 1 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north1
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north1-tx.pcap], [vif-north1.expected])
> +
> +# N-S with lp-north2
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 10 0 0 10`
> +tip=`ip_to_hex 10 0 0 50`
> +test_arp vif-north2 f0f000000022 $sip $tip
> +
> +echo "Send traffic South to North2"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 10 0 0 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north2
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north2-tx.pcap], [vif-north2.expected])
> +
> +# N-S with lp-north3
> +echo "Send Dummy ARP"
> +sip=`ip_to_hex 192 168 0 10`
> +tip=`ip_to_hex 192 168 0 50`
> +test_arp vif-north3 f0f000000033 $sip $tip
> +
> +echo "Send traffic South to North3"
> +sip=`ip_to_hex 20 0 0 10`
> +dip=`ip_to_hex 192 168 0 10`
> +test_ip vif11 f00000000010 06ac10010001 $sip $dip vif-north3
> +# Confirm that South to North traffic works fine.
> +OVN_CHECK_PACKETS_REMOVE_BROADCAST([hv5/vif-north3-tx.pcap], [vif-north3.expected])
> +
> +AT_CLEANUP
> +])
> --
> 2.30.2
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
More information about the dev
mailing list