[ovs-discuss] Traffic loss after TEP deletion in OVS

Balazs Nemeth balazs.nemeth at ericsson.com
Mon Sep 19 15:53:52 UTC 2016


Dear All,

I think we found a bug in OVS master. In the test case 3 OVS is used, VXLAN tunnels are configured among them in full mesh, so every node has 2 Tunnel End Point (TEP). BFD is turned on for monitoring the tunnel liveness. In the beginning, traffic and BFD messages are ongoing between all the nodes. If you delete 1 TEP from node-1 (TEP13 towards node-3), then the remaining TEP on node-1 (TEP12 towards node-2) will be affected also. I expect it to work after deleting another TEP, but it does not transmit packets. It will not be able to terminate VXLAN encapsulated packets any more! Due to this the BFD Forwarding status of the remaining TEP will go to False, and traffic will be dropped between this TEP and the another TEP of this remaining tunnel. I also see that dpctl flows for remaining tunnel will disappear after 10 sec.

I think when you delete one TEP from the two, some data or setting of the remaining TEP will be overwritten improperly. Due to RFC 7348, OVS will accept VXLAN encapsulated packets on UDP dst_port 4789 by default. Perhaps this dst_port value will be overwritten. I made a port-mirroring on the physical interface and I can see packets like:
d6:ee:ac:b9:6c:81 > 00:23:20:00:00:01, ethertype IPv4 (0x0800), length 66: 169.254.1.1.49186 > 169.254.1.0.3784: BFDv1, Control, State Down, Flags: [none], length: 24
a0:36:9f:43:f2:f8 > a0:36:9f:43:f3:d8, ethertype IPv4 (0x0800), length 144: 10.85.46.4 > 10.85.46.7: ICMP 10.85.46.4 udp port 4789 unreachable, length 110

The issue can be fixed if you change any parameter of the remaining TEP. After that TEP maybe reinitialized, and traffic can go through (BFD Forwarding will be True again), e.g. chaging remote_ip to random value and back:

1.       node-1: ovs-vsctl set Interface TEP12 options:remote_ip=10.85.46.254

node-1: ovs-vsctl set Interface TEP12 options:remote_ip=10.85.46.7
The issue can be fixed also e.g. with modifying dst_port of TEPs between node-1 and node-2:

2.  node-1: ovs-vsctl set Interface TEP12 options:dst_port=4790

node-2: ovs-vsctl set Interface TEP21 options:dst_port=4790

We are working on the fix, any help or tips would be appreciated.

Details:
root at node-1:~# ovs-vswitchd --version
ovs-vswitchd (Open vSwitch) 2.6.90
user at node-1:~/openvswitch(master)> git rev-parse HEAD
75e2077e0c43224bcca92746b28b01a4936fc101
user at node-1:~/dpdk((v16.07))<mailto:user at node-1:~/dpdk((v16.07))>> git rev-parse HEAD
20e2b6eba13d9eb61b23ea75f09f2aa966fa6325
No local change or patch. OS: Ubuntu 14.04.4, kernel: 3.13.0-79-generic, no OF controller.
We are running OVS latest master with DPDK 16.07 datapath. TEPs are added to netdev bridges, like:
root at node-1:~# ovs-vsctl add-port br-int TEP12 -- set Interface TEP12 type=vxlan options:key=flow options:local_ip=10.85.46.4 options:remote_ip=10.85.46.7 bfd:enable=true bfd:min_tx=1000 ofport_request=12
Example OVS bridge structure:
root at node-1:~# ovs-vsctl show
46ffe13d-fdf0-4bc7-b71e-5d0909032c94
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port "TEP12"
            Interface "TEP12"
                type: vxlan
                options: {key=flow, local_ip="10.85.46.4", remote_ip="10.85.46.7"}
        Port "TEP13"
            Interface "TEP13"
                type: vxlan
                options: {key=flow, local_ip="10.85.46.4", remote_ip="10.85.46.6"}
    Bridge br-prv
        Port br-prv
            Interface br-prv
                type: internal
        Port bond-prv
            Interface "dpdk0"
                type: dpdk
            Interface "dpdk1"
                type: dpdk
    ovs_version: "2.6.90"

BR,
Balazs Nemeth
Ericsson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160919/6aeda55d/attachment-0002.html>


More information about the discuss mailing list