[ovs-discuss] [OVN] Packets flooded when using VLAN backed networks

Thu Sep 17 16:53:16 UTC 2020

Hi Daniel,

Glad to see your reply.
Yes, we need to do something about it. I will try to come back with some patches (or IRC discussions) and we will discuss the same further.

Thanks

Regards,
Ankur
________________________________
From: Daniel Alvarez Sanchez <dalvarez at redhat.com>
Sent: Thursday, September 17, 2020 3:11 AM
To: Ankur Sharma <ankur.sharma at nutanix.com>
Cc: ovs-discuss <ovs-discuss at openvswitch.org>
Subject: Re: [OVN] Packets flooded when using VLAN backed networks

Thanks Ankur for your reply!

On Sat, Sep 12, 2020 at 4:27 AM Ankur Sharma <ankur.sharma at nutanix.com<mailto:ankur.sharma at nutanix.com>> wrote:
Hi Daniel,

Thanks a lot for starting the thread.
Yes, you have a valid observation.

"The reason is that, as we translate the eth.src to that of the "ovn-chassis-mac-mappings", the ToR will never see a packet whose eth.src is either vm1 or vm3 so it'll never learn their addresses and flood the traffic to all ports."

The main reason is not chassis-mac-mappings, but the way OVN LR works.

Yes, that's right! Sorry for the confusion. Even with the 'reside-on-redirect-chassis' option we'd be observing this as well.

TOPOLOGY:
LS1 ---- LR ----LS2

VM1--LS1---CHASSIS1
VM2--LS2---CHASSIS2

Let us say we are pinging from VM1 --> VM2.

a. For a routed traffic , we will never have source mac as VM mac, i.e once traffic is routed then, source mac will be replaced.
b. The reason flooding is observed is because of following:
    i. A typical router will send ARP for the destination endpoint (VM2 in the example above) and the corresponding reply will cause initial learning of VM MAC (VM2 mac in the example above).
   ii. Similarly, after initial ARP resolution, it will do periodic ARP refresh by generating an ARP request (could be a unicast ARP) and again the reply will ensure that MAC entry in TOR is refreshed.
c. Now, OVN LR may not send out ARP request on the wire (because of ARP suppression) and even if it does (let us say port just has mac and not the IP), then ARP response is not sent on the wire.
d. Similarly, since there is no periodic ARP refresh, hence even if somehow initial learning happens, the MAC will eventually age out.

We can do either of following:
a. Assign a chassis to do periodic ARP refreshes.
    i. I started some work around this, but looked tricky, i.e besides sending the ARP refresh, then we should also remove the ARP cache entry for which certain threshhold of ARP refresh is failing.
b. Periodic RARP advertisement of endpoints on localnet logical switches.

And like you said, we can use this thread to converge.

Thanks Ankur!  I don't have a strong opinion of either of the above, but it looks like we want to do something about this (maybe configurable) to avoid situations where we flood large amounts of traffic (for example in long lived connections or bulk transfers or ...)

Thanks

Regards,
Ankur

________________________________
From: Daniel Alvarez Sanchez <dalvarez at redhat.com<mailto:dalvarez at redhat.com>>
Sent: Friday, September 11, 2020 8:57 AM
To: ovs-discuss <ovs-discuss at openvswitch.org<mailto:ovs-discuss at openvswitch.org>>
Cc: Ankur Sharma <ankur.sharma at nutanix.com<mailto:ankur.sharma at nutanix.com>>
Subject: [OVN] Packets flooded when using VLAN backed networks

Hi folks,

This is probably not a bug and not sure if much can be done about it but thought of raising it here for discussion.

I have deployed a simple topology with two logical switches (VLAN backed network), a logical router and a couple of VMs. When pinging between the logical switches, all the traffic is flooded in the upstream switch.

Example using this logical [0] and physical [1] topologies, when pinging from vm3 (worker2) to vm1 (worker1) and capturing traffic on host1:

15:50:18.790323 1e:02:ad:bb:aa:dd > 40:44:00:00:00:01, ethertype 802.1Q (0x8100), length 102: vlan 190, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 47366, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.1.13 > 192.168.0.11 [192.168.0.11]<https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.0.11&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=mZwX9gFQgeJHzTg-68aCJgsODyUEVsHGFOfL90J6MJY&m=ismplsHTP5PsuM6SoMp5RKExr_6JaJ_jhFin7Bf622Y&s=b3pOCoAucQQ2VcKZ4gJOCmHfPfiiyyrhh7miaE-a7z8&e=>: ICMP echo request, id 1671, seq 11, length 64
15:50:18.790428 1e:02:ad:bb:aa:77 > 40:44:33:00:00:03, ethertype 802.1Q (0x8100), length 102: vlan 170, p 0, ethertype IPv4, (tos 0x0, ttl 63, id 44948, offset 0, flags [none], proto ICMP (1), length 84)
    192.168.0.11 > 192.168.1.13 [192.168.1.13]<https://urldefense.proofpoint.com/v2/url?u=http-3A__192.168.1.13&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=mZwX9gFQgeJHzTg-68aCJgsODyUEVsHGFOfL90J6MJY&m=ismplsHTP5PsuM6SoMp5RKExr_6JaJ_jhFin7Bf622Y&s=bySzVdejw7-0AUNDmSi5lKdOcBsMpeeRSrXioKAZt48&e=>: ICMP echo reply, id 1671, seq 11, length 64

The reason is that, as we translate the eth.src to that of the "ovn-chassis-mac-mappings", the ToR will never see a packet whose eth.src is either vm1 or vm3 so it'll never learn their addresses and flood the traffic to all ports.

In the example above:

[root at worker1 ~]# ovs-vsctl get open . external_ids:ovn-chassis-mac-mappings
"tenant:1e:02:ad:bb:aa:77"

[root at worker2 vagrant]# ovs-vsctl get open . external_ids:ovn-chassis-mac-mappings
"tenant:1e:02:ad:bb:aa:dd"

I understand that the benefit of using the ovn-chassis-mac-mappings is the distributed routing capabilities but I wonder if we could come up with a way of avoiding the flood.

In case somebody's interested in replicating this scenario, you can find a vagrant setup here [2].

Thanks!
daniel

[0] http://dani.foroselectronica.es/wp-content/uploads/2020/09/extp_log.png [dani.foroselectronica.es]<https://urldefense.proofpoint.com/v2/url?u=http-3A__dani.foroselectronica.es_wp-2Dcontent_uploads_2020_09_extp-5Flog.png&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=mZwX9gFQgeJHzTg-68aCJgsODyUEVsHGFOfL90J6MJY&m=ismplsHTP5PsuM6SoMp5RKExr_6JaJ_jhFin7Bf622Y&s=m97u9m3BWAD7xAFeU0x8IJ0C8UmIw-BmGyJOfnqV5mQ&e=>
[1] http://dani.foroselectronica.es/wp-content/uploads/2020/09/expt_phy.png [dani.foroselectronica.es]<https://urldefense.proofpoint.com/v2/url?u=http-3A__dani.foroselectronica.es_wp-2Dcontent_uploads_2020_09_expt-5Fphy.png&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=mZwX9gFQgeJHzTg-68aCJgsODyUEVsHGFOfL90J6MJY&m=ismplsHTP5PsuM6SoMp5RKExr_6JaJ_jhFin7Bf622Y&s=T3Bao1_MJaNNctCJ0l8CZusfFFwiHJw442MIg9TvFUE&e=>
[2] https://github.com/danalsan/vagrants/tree/master/ovn-external-ports [github.com]<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_danalsan_vagrants_tree_master_ovn-2Dexternal-2Dports&d=DwMFaQ&c=s883GpUCOChKOHiocYtGcg&r=mZwX9gFQgeJHzTg-68aCJgsODyUEVsHGFOfL90J6MJY&m=ismplsHTP5PsuM6SoMp5RKExr_6JaJ_jhFin7Bf622Y&s=-oaANnssk2VukewpNIbVsSzd9AgkGVuqDj-NAG5q3rY&e=>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200917/599bfed5/attachment.html>