[ovs-discuss] [External] : Re: Is this a bug in the "Egress Loopback table" or am I missing something

Brendan Doyle brendan.doyle at oracle.com
Thu Jul 1 09:55:44 UTC 2021



On 30/06/2021 18:44, Numan Siddique wrote:
> On Wed, Jun 30, 2021 at 10:54 AM Brendan Doyle <brendan.doyle at oracle.com> wrote:
>>
>> OK so the simple 1 line change to northd.c in:
>>
>>    [ovs-dev,v8,1/6] northd: Swap src and dst eth addresses in router
>> egress loop.
>>
>> fixes the problem, can access all external networks, and the haripin
>> between 10.68.49.185 <-> 10.68.49.185
>> works. Thumbs up for me on this patch!
>>
> Thanks for testing out.  I applied that patch to the main branch and
> backported to branch-21.06.
>
> Thanks
> Numan

Great - thanks
>> On 30/06/2021 10:11, Brendan Doyle wrote:
>>> So If I do :
>>>
>>> ovn-nbctl add logical_router_port lr1-ls1_external networks
>>> "10.68.49.185/32 10.68.49.184/32"
>>>
>>> Then the hairpin works and I have connectivity between 10.68.49.185
>>> <-> 10.68.49.185
>>>
>>> But This patch also look promising:
>>> [ovs-dev,v8,1/6] northd: Swap src and dst eth addresses in router
>>> egress loop.
>>>
>>> I'll try adding this, and incrementally the other patches in the series.
>>>
>>> Brendan
>>>
>>>
>>> On 29/06/2021 22:40, Numan Siddique wrote:
>>>> On Tue, Jun 29, 2021 at 5:06 PM Brendan Doyle
>>>> <brendan.doyle at oracle.com> wrote:
>>>>>
>>>>> On 29/06/2021 21:38, Numan Siddique wrote:
>>>>>> On Tue, Jun 29, 2021 at 4:13 PM Brendan Doyle
>>>>>> <brendan.doyle at oracle.com> wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> With a very simple notwork (two VMs on different chassis), 1 subnet,
>>>>>>> single LS and
>>>>>>> LR/Gateway. The two VMs can ping each other using their Logical IPs.
>>>>>>> Each has an
>>>>>>> "External IP", and each can be accessed from an external network
>>>>>>> on that
>>>>>>> external IP.
>>>>>>> BUT they can't ping each other using their external IPs. I would have
>>>>>>> expected that
>>>>>>> either:
>>>>>>>
>>>>>>> a) The packets are sent on the external net then hairpinned back
>>>>>>> to the OVN
>>>>>>>          gateway by the external net router.
>>>>>>>
>>>>>>> b) They are hairpinned by OVN.
>>>>>>>
>>>>>>> It seems that OVN attempts the latter, but does not succeed. The
>>>>>>> details, NB network,
>>>>>>> and pkt trace are as follows:
>>>>>>>
>>>>>>> ovn-nbctl show
>>>>>>> switch 2710eebe-f2b3-49e4-bcd6-dcfa48ed6470 (ls1_external)
>>>>>>>         port ln-ls1_external
>>>>>>>             type: localnet
>>>>>>>             addresses: ["unknown"]
>>>>>>>         port ls1_external-lr1
>>>>>>>             type: router
>>>>>>>             router-port: lr1-ls1_external
>>>>>>>
>>>>>>> switch ff909b16-d863-4e3d-a10b-2f0010f17b23 (ls1)
>>>>>>>         port 47433b54-ac10-42f1-ae84-cc6fbb580297
>>>>>>>             addresses: ["52:54:00:be:06:16 192.16.1.6"]
>>>>>>>         port 00bff7c0-2e2d-41ba-9485-3b5fa9801365
>>>>>>>             addresses: ["52:54:00:e6:4f:46 192.16.1.5"]
>>>>>>>         port ls1-lr1
>>>>>>>             type: router
>>>>>>>             router-port: lr1-ls1
>>>>>>>
>>>>>>> router 63e1b6a2-327f-4a24-b0c9-3a0e951beb2b (lr1)
>>>>>>>         port lr1-ls1_external
>>>>>>>             mac: "40:44:00:00:01:a0"
>>>>>>>             networks: ["253.255.80.10/16"]
>>>>>>>             gateway chassis: [ca-rain06 ca-rain17 ca-rain05]
>>>>>>>         port lr1-ls1
>>>>>>>             mac: "40:44:00:00:01:30"
>>>>>>>             networks: ["192.16.1.1/24"]
>>>>>>>         nat f4675661-f4cc-4f7c-b534-ca75e090ed74
>>>>>>>             external ip: "10.68.49.184"
>>>>>>>             logical ip: "192.16.1.5"
>>>>>>>             type: "dnat_and_snat"
>>>>>>>         nat f5592262-5fbd-4cef-8773-903875ba34d6
>>>>>>>             external ip: "10.68.49.185"
>>>>>>>             logical ip: "192.16.1.6"
>>>>>>>             type: "dnat_and_snat"
>>>>>>>
>>>>>> Why don't the external ips belong to the subnet - 253.255.80.10/16 ?
>>>>>> i.e to the network of ls1_external ?
>>>>> The 253.255.80.10/16 network is an internal "underlay" Network. An
>>>>> infra
>>>>> structure network
>>>>> of the rack product. The "External IPs", are IPs belonging to networks
>>>>> outside the rack.
>>>>>
>>>>> So in Normal case traffic  destined for a VM from outside the rack,
>>>>> would send to the VM
>>>>> "External IP", that arrives at the rack physical uplink router, and is
>>>>> sent across the rack
>>>>> physical network (253.255.0.0/16) to the OVN Gateway, which DNATs and
>>>>> send to the VM
>>>>> Logical IP (reverse on traffic from VM to destination outside the
>>>>> rack).
>>>>>
>>>>>
>>>>>> I'm pretty sure if you change the external_ips from 10.68.49.184 and
>>>>>> 10.68.49.185 to
>>>>>> the ones belonging to 253.255.80.10/16, it would work.
>>>>> We can't do that, these are different address spaces in different
>>>>> physical networks.
>>>>> I could try adding the 10.68.49.184/185 IPs to the "networks" table in
>>>>> lr1-ls1_external
>>>>>> I'd suggest trying out with these patches once ? -
>>>>>> https://urldefense.com/v3/__https://patchwork.ozlabs.org/project/ovn/list/?series=247106__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIiqu4xmH8$
>>>>>>
>>>>> Ok, will do, are they in master, as I'm running with a fairly recent
>>>>> build (maybe two weeks old)
>>>> The patches are still under review and may not apply cleanly with the
>>>> tip.  You can access it from here too -
>>>> https://urldefense.com/v3/__https://github.com/ovsrobot/ovn/commits/series_247106__;!!ACWV5N9M2RV99hQ!e3YISaySCgi6qg3Y-8_gdx0IN_FeVsl5onOgkxhBhhgp_69r8PTAROpeu3yG3eaPN0c$
>>>>
>>>>
>>>> Thanks
>>>> Numan
>>>>
>>>>> Thanks
>>>>>
>>>>>> Numan
>>>>>>
>>>>>>
>>>>>>> ovn-nbctl lr-route-list lr1
>>>>>>> IPv4 Routes
>>>>>>>                     0.0.0.0/0               253.255.0.1 dst-ip
>>>>>>> lr1-ls1_external
>>>>>>>
>>>>>>> ovn-trace --detailed ls1 'inport ==
>>>>>>> "47433b54-ac10-42f1-ae84-cc6fbb580297" && eth.dst ==
>>>>>>> 40:44:00:00:01:30
>>>>>>> && eth.src == 52:54:00:be:06:16 && ip4.src == 192.16.1.6 &&
>>>>>>> ip4.dst ==
>>>>>>> 10.68.49.184 && ip.ttl == 64 && icmp4.type == 8'
>>>>>>> #
>>>>>>> icmp,reg14=0x1,vlan_tci=0x0000,dl_src=52:54:00:be:06:16,dl_dst=40:44:00:00:01:30,nw_src=192.16.1.6,nw_dst=10.68.49.184,nw_tos=0,nw_ecn=0,nw_ttl=64,icmp_type=8,icmp_code=0
>>>>>>>
>>>>>>>
>>>>>>> ingress(dp="ls1", inport="47433b")
>>>>>>> ----------------------------------
>>>>>>>      0. ls_in_port_sec_l2 (ovn-northd.c:4834): inport == "47433b",
>>>>>>> priority
>>>>>>> 50, uuid ae50c799
>>>>>>>         next;
>>>>>>> 22. ls_in_l2_lkup (ovn-northd.c:7587): eth.dst == 40:44:00:00:01:30,
>>>>>>> priority 50, uuid c29dec2a
>>>>>>>         outport = "ls1-lr1";
>>>>>>>         output;
>>>>>>>
>>>>>>> egress(dp="ls1", inport="47433b", outport="ls1-lr1")
>>>>>>> ----------------------------------------------------
>>>>>>>      0. ls_out_pre_lb (ovn-northd.c:4980): ip && outport == "ls1-lr1",
>>>>>>> priority 110, uuid d4d7c7af
>>>>>>>         next;
>>>>>>>      9. ls_out_port_sec_l2 (ovn-northd.c:4929): outport == "ls1-lr1",
>>>>>>> priority 50, uuid 36b335f9
>>>>>>>         output;
>>>>>>>         /* output to "ls1-lr1", type "patch" */
>>>>>>>
>>>>>>> ingress(dp="lr1", inport="lr1-ls1")
>>>>>>> -----------------------------------
>>>>>>>      0. lr_in_admission (ovn-northd.c:9575): eth.dst ==
>>>>>>> 40:44:00:00:01:30
>>>>>>> && inport == "lr1-ls1", priority 50, uuid c67387d7
>>>>>>>         xreg0[0..47] = 40:44:00:00:01:30;
>>>>>>>         next;
>>>>>>>      1. lr_in_lookup_neighbor (ovn-northd.c:9654): 1, priority 0,
>>>>>>> uuid c050ede1
>>>>>>>         reg9[2] = 1;
>>>>>>>         next;
>>>>>>>      2. lr_in_learn_neighbor (ovn-northd.c:9663): reg9[2] == 1,
>>>>>>> priority
>>>>>>> 100, uuid e5780577
>>>>>>>         next;
>>>>>>> 10. lr_in_ip_routing (ovn-northd.c:8622): ip4.dst == 0.0.0.0/0,
>>>>>>> priority
>>>>>>> 1, uuid 52d001c6
>>>>>>>         ip.ttl--;
>>>>>>>         reg8[0..15] = 0;
>>>>>>>         reg0 = 253.255.0.1;
>>>>>>>         reg1 = 253.255.80.10;
>>>>>>>         eth.src = 40:44:00:00:01:a0;
>>>>>>>         outport = "lr1-ls1_external";
>>>>>>>         flags.loopback = 1;
>>>>>>>         next;
>>>>>>> 11. lr_in_ip_routing_ecmp (ovn-northd.c:9921): reg8[0..15] == 0,
>>>>>>> priority 150, uuid 920ee40c
>>>>>>>         next;
>>>>>>> 12. lr_in_policy (ovn-northd.c:10046): 1, priority 0, uuid e2014343
>>>>>>>         reg8[0..15] = 0;
>>>>>>>         next;
>>>>>>> 13. lr_in_policy_ecmp (ovn-northd.c:10048): reg8[0..15] == 0,
>>>>>>> priority
>>>>>>> 150, uuid ed8c4d4d
>>>>>>>         next;
>>>>>>> 14. lr_in_arp_resolve (ovn-northd.c:10082): ip4, priority 0, uuid
>>>>>>> 2cfde30a
>>>>>>>         get_arp(outport, reg0);
>>>>>>>         /* MAC binding to 00:00:0c:07:ac:14. */
>>>>>>>         next;
>>>>>>> 17. lr_in_gw_redirect (ovn-northd.c:10598): outport ==
>>>>>>> "lr1-ls1_external", priority 50, uuid 521a9223
>>>>>>>         outport = "cr-lr1-ls1_external";
>>>>>>>         next;
>>>>>>> 18. lr_in_arp_request (ovn-northd.c:10671): 1, priority 0, uuid
>>>>>>> e43fdfbd
>>>>>>>         output;
>>>>>>>         /* Replacing type "chassisredirect" outport
>>>>>>> "cr-lr1-ls1_external"
>>>>>>> with distributed port "lr1-ls1_external". */
>>>>>>>
>>>>>>> egress(dp="lr1", inport="lr1-ls1", outport="lr1-ls1_external")
>>>>>>> --------------------------------------------------------------
>>>>>>>      0. lr_out_undnat (ovn-northd.c:11459): ip && ip4.src ==
>>>>>>> 192.16.1.6 &&
>>>>>>> outport == "lr1-ls1_external" &&
>>>>>>> is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid
>>>>>>> e8b081df
>>>>>>>         ct_dnat;
>>>>>>>
>>>>>>> ct_dnat /* assuming no un-dnat entry, so no change */
>>>>>>> -----------------------------------------------------
>>>>>>>      1. lr_out_snat (ovn-northd.c:11552): ip && ip4.src ==
>>>>>>> 192.16.1.6 &&
>>>>>>> outport == "lr1-ls1_external" &&
>>>>>>> is_chassis_resident("cr-lr1-ls1_external"), priority 161, uuid
>>>>>>> f50e5215
>>>>>>>         ct_snat(10.68.49.185);
>>>>>>>
>>>>>>> ct_snat(ip4.src=10.68.49.185)
>>>>>>> -----------------------------
>>>>>>>      2. lr_out_egr_loop (ovn-northd.c:11846): ip4.dst ==
>>>>>>> 10.68.49.184 &&
>>>>>>> outport == "lr1-ls1_external" &&
>>>>>>> is_chassis_resident("cr-lr1-ls1_external"), priority 100, uuid
>>>>>>> a6499050
>>>>>>>         clone { ct_clear; inport = outport; outport = ""; flags = 0;
>>>>>>> flags.loopback = 1; reg0 = 0; reg1 = 0; reg2 = 0; reg3 = 0; reg4 = 0;
>>>>>>> reg5 = 0; reg6 = 0; reg7 = 0; reg8 = 0; reg9 = 0; reg9[0] = 1;
>>>>>>> next(pipeline=ingress, table=0); };
>>>>>>>
>>>>>>> clone
>>>>>>> -----
>>>>>>>         ct_clear;
>>>>>>>         inport = outport;
>>>>>>>         outport = "";
>>>>>>>         flags = 0;
>>>>>>>         flags.loopback = 1;
>>>>>>>         reg0 = 0;
>>>>>>>         reg1 = 0;
>>>>>>>         reg2 = 0;
>>>>>>>         reg3 = 0;
>>>>>>>         reg4 = 0;
>>>>>>>         reg5 = 0;
>>>>>>>         reg6 = 0;
>>>>>>>         reg7 = 0;
>>>>>>>         reg8 = 0;
>>>>>>>         reg9 = 0;
>>>>>>>         reg9[0] = 1;
>>>>>>>         next(pipeline=ingress, table=0);
>>>>>>>
>>>>>>> ingress(dp="lr1", inport="lr1-ls1_external")
>>>>>>> --------------------------------------------
>>>>>>>      0. lr_in_admission: no match (implicit drop)
>>>>>>>
>>>>>>> If we look at the section of code pointed to by ovn-northd.c:11846
>>>>>>>
>>>>>>>            /* Egress Loopback table: For NAT on a distributed router.
>>>>>>>              * If packets in the egress pipeline on the distributed
>>>>>>>              * gateway port have ip.dst matching a NAT external IP,
>>>>>>> then
>>>>>>>              * loop a clone of the packet back to the beginning of the
>>>>>>>              * ingress pipeline with inport = outport. */
>>>>>>>             if (od->l3dgw_port) {
>>>>>>>                 /* Distributed router. */
>>>>>>>                 ds_clear(match);
>>>>>>>                 ds_put_format(match, "ip%s.dst == %s && outport ==
>>>>>>> %s",
>>>>>>>                               is_v6 ? "6" : "4",
>>>>>>>                               nat->external_ip,
>>>>>>> od->l3dgw_port->json_key);
>>>>>>>                 if (!distributed) {
>>>>>>>                     ds_put_format(match, " &&
>>>>>>> is_chassis_resident(%s)",
>>>>>>> od->l3redirect_port->json_key);
>>>>>>>                 } else {
>>>>>>>                     ds_put_format(match, " &&
>>>>>>> is_chassis_resident(\"%s\")",
>>>>>>>                                   nat->logical_port);
>>>>>>>                 }
>>>>>>>                ds_clear(actions);
>>>>>>>                 ds_put_format(actions,
>>>>>>>                               "clone { ct_clear; "
>>>>>>>                               "inport = outport; outport = \"\"; "
>>>>>>>                               "flags = 0; flags.loopback = 1; ");
>>>>>>>                 for (int j = 0; j < MFF_N_LOG_REGS; j++) {
>>>>>>>                     ds_put_format(actions, "reg%d = 0; ", j);
>>>>>>>                 }
>>>>>>>                 ds_put_format(actions, REGBIT_EGRESS_LOOPBACK" = 1; "
>>>>>>>                               "next(pipeline=ingress, table=%d); };",
>>>>>>> ovn_stage_get_table(S_ROUTER_IN_ADMISSION));
>>>>>>>                 ovn_lflow_add_with_hint(lflows, od,
>>>>>>> S_ROUTER_OUT_EGR_LOOP, 100,
>>>>>>>                                         ds_cstr(match),
>>>>>>> ds_cstr(actions),
>>>>>>> &nat->header_);
>>>>>>>             }
>>>>>>>
>>>>>>> It seems clear what the intent is, but the pkt is dropped immediately
>>>>>>> when returned to the ingress
>>>>>>> pipeline. Am I missing some config?
>>>>>>>
>>>>>>>
>>>>>>> Thanks Brendan
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> discuss mailing list
>>>>>>> discuss at openvswitch.org
>>>>>>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!ZKO2z-ifCaUA-TPeLm7ZP9V7hkX8tZSv4HE4-Ogo2BhBcLfSbibLIh4xDsIi0Md7RaE$
>>>>>>>
>>>>> _______________________________________________
>>>>> discuss mailing list
>>>>> discuss at openvswitch.org
>>>>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!e3YISaySCgi6qg3Y-8_gdx0IN_FeVsl5onOgkxhBhhgp_69r8PTAROpeu3yGTHgIUMg$
>>>>>
>>> _______________________________________________
>>> discuss mailing list
>>> discuss at openvswitch.org
>>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!Z3k6-fDQJqwNn8Agn4ngwEJJy4wHaYT_i3fUlmeofW97TlsXsnMU3UMjNkbzwhW9YBM$
>>
>> _______________________________________________
>> discuss mailing list
>> discuss at openvswitch.org
>> https://urldefense.com/v3/__https://mail.openvswitch.org/mailman/listinfo/ovs-discuss__;!!ACWV5N9M2RV99hQ!eHGRvvWJ6ZN7cpnVsXnoCfFwJBkjvn3WJ4Flvg7Yuu55FZrCotgGVeyHzLl2a_fbUhQ$



More information about the discuss mailing list