[ovs-discuss] OVS lost packages with source ip from external network

Sergey Yezhkov yezhkov at gmail.com
Tue Dec 3 16:00:02 UTC 2019


Hi,

I have environment with OpenStack and OVS as switch and time to time OVS lost packages.
It hard to reproduce - that’s why I will describe my real environment.

I have bridge br-int with ports:
- qr-6fe81a87-ae - with configured ip 10.1.0.129 and mac fa:16:3e:82:3f:9b (this port created inside namespace )
- qvo532c9b8b-da - connected to VM with ip 10.1.0.174 and mac fa:16:3e:f2:e3:1d

ovs-vsctl show
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: system
        Port br-int
            Interface br-int
                type: internal
- - -
        Port "qr-6fe81a87-ae"
            tag: 19
            Interface "qr-6fe81a87-ae"
                type: internal
- - -
        Port "qvo532c9b8b-da"
            tag: 19
            Interface "qvo532c9b8b-da”

Both ports in same VLAN 19

And I have another port br_pub with ip 172.17.154.20 and iptables with DNAT rule
'DNAT       all  --  anywhere             172.17.154.154       to:10.1.0.176'

What the problem:
when i ping from 10.1.0.129 (port qr-6fe81a87-ae) it work stable
but when i ping from external IP
ping -i br_pub 172.17.154.154 - some time work, some time not.
It can work half an hour then do not work next half an hour.

When it NOT works:
ovs-tcpdump -i qr-6fe81a87-ae -lnne icmp
13:34:53.697386 fa:16:3e:82:3f:9b > fa:16:3e:f2:e3:1d, ethertype 802.1Q (0x8100), length 102: vlan 19, p 0, ethertype IPv4, 172.17.154.20 > 10.1.0.176: ICMP echo request, id 60485, seq 1, length 64

ovs-tcpdump -i qvo532c9b8b-da -lnne icmp
no output

That’s I decide that problem inside OVS

When it works:
I see packets on both ports

Expected behavior:
I should see packets on both ports stable the same as when ping from 10.1.0.129

I dump flows on br-int when it work and when it not work - there is no difference.

After reboot ovs-vswitchd it start to work properly for some time.

Some details:
- ovs-vswitchd —version
ovs-vswitchd (Open vSwitch) 2.12.0

- cat /proc/version
Linux version 4.15.0-70-generic (buildd at lgw01-amd64-057) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)) #79~16.04.1-Ubuntu SMP Tue Nov 12 14:01:10 UTC 2019

In attachment conf.db, ovs-vsctl-show.txt, ovs-ofctl-dump-flows.txt

ovs-appctl fdb/show br-int | grep "fa:16:3e:82:3f:9b”
  262    19  fa:16:3e:82:3f:9b    0
Source port #262
ovs-appctl fdb/show br-int | grep "fa:16:3e:f2:e3:1d"
  228    19  fa:16:3e:f2:e3:1d    0
Destanation port #228

ovs-appctl ofproto/trace br-int in_port=262,icmp,dl_vlan=19,dl_dst=fa:16:3e:f2:e3:1d
Flow: icmp,in_port=262,dl_vlan=19,dl_vlan_pcp=0,vlan_tci1=0x0000,dl_src=00:00:00:00:00:00,dl_dst=fa:16:3e:f2:e3:1d,nw_src=0.0.0.0,nw_dst=0.0.0.0,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0

bridge("br-int")
----------------
 0. priority 0, cookie 0x8729459f5bc74e9a
    goto_table:60
60. dl_vlan=19,dl_dst=fa:16:3e:f2:e3:1d, priority 4, cookie 0x8729459f5bc74e9a
    pop_vlan
    output:228

Final flow: icmp,in_port=262,vlan_tci=0x0000,dl_src=00:00:00:00:00:00,dl_dst=fa:16:3e:f2:e3:1d,nw_src=0.0.0.0,nw_dst=0.0.0.0,nw_tos=0,nw_ecn=0,nw_ttl=0,icmp_type=0,icmp_code=0
Megaflow: recirc_id=0,eth,ip,in_port=262,dl_vlan=19,dl_vlan_pcp=0,dl_dst=fa:16:3e:f2:e3:1d,nw_frag=no
Datapath actions: pop_vlan,18

ovs-tcpdump -i qr-6fe81a87-ae -lnne icmp
13:34:47.673408 fa:16:3e:82:3f:9b > fa:16:3e:f2:e3:1d, ethertype 802.1Q (0x8100), length 102: vlan 19, p 0, ethertype IPv4, 10.1.0.129 > 10.1.0.176: ICMP echo request, id 60484, seq 1, length 64
13:34:47.674144 fa:16:3e:f2:e3:1d > fa:16:3e:82:3f:9b, ethertype 802.1Q (0x8100), length 102: vlan 19, p 0, ethertype IPv4, 10.1.0.176 > 10.1.0.129: ICMP echo reply, id 60484, seq 1, length 64
13:34:53.697386 fa:16:3e:82:3f:9b > fa:16:3e:f2:e3:1d, ethertype 802.1Q (0x8100), length 102: vlan 19, p 0, ethertype IPv4, 172.17.154.20 > 10.1.0.176: ICMP echo request, id 60485, seq 1, length 64

ovs-tcpdump -i qvo532c9b8b-da -lnne icmp
13:33:24.953624 fa:16:3e:82:3f:9b > fa:16:3e:f2:e3:1d, ethertype 802.1Q (0x8100), length 102: vlan 19, p 0, ethertype IPv4, 10.1.0.129 > 10.1.0.176: ICMP echo request, id 59708, seq 1, length 64
13:33:24.954159 fa:16:3e:f2:e3:1d > fa:16:3e:82:3f:9b, ethertype 802.1Q (0x8100), length 102: vlan 19, p 0, ethertype IPv4, 10.1.0.176 > 10.1.0.129: ICMP echo reply, id 59708, seq 1, length 64
and no icmp request with source ip 172.17.154.20

Some details about environment physical server os-comp00 (openstack compute host) with:
- port br_pub - ip 172.17.154.20
- net namespace qrouter-6b7abf6b-7391-4b6e-8829-35d7aa448761 with port qr-6fe81a87-ae with ip 10.1.0.129 and mac fa:16:3e:82:3f:9b
- OVS deployed by docker containers:
* openvswitch_db with command '/usr/sbin/ovsdb-server /var/lib/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --remote=ptcp:6640:127.0.0.1 --remote=db:Open_vSwitch,Open_vSwitch,manager_options --log-file=/var/log/kolla/openvswitch/ovsdb-server.log --pidfile’
* openvswitch_vswitch with command '/usr/sbin/ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log —pidfile’
- instance with tap port tap532c9b8b-da connected through linux bridge to veth patch qvb532c9b8b-da at qvo532c9b8b-da

root at os-comp00:~# ip netns exec qrouter-6b7abf6b-7391-4b6e-8829-35d7aa448761 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: rfp-6b7abf6b-7 at if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 42:b4:ba:34:33:5f brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.127.96/31 scope global rfp-6b7abf6b-7
       valid_lft forever preferred_lft forever
    inet6 fe80::40b4:baff:fe34:335f/64 scope link 
       valid_lft forever preferred_lft forever
834: qr-6fe81a87-ae: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:82:3f:9b brd ff:ff:ff:ff:ff:ff
    inet 10.1.0.129/26 brd 10.1.0.191 scope global qr-6fe81a87-ae
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe82:3f9b/64 scope link 
       valid_lft forever preferred_lft forever

root at os-comp00:~# ip netns exec qrouter-6b7abf6b-7391-4b6e-8829-35d7aa448761 ip route
10.1.0.128/26 dev qr-6fe81a87-ae  proto kernel  scope link  src 10.1.0.129 
169.254.127.96/31 dev rfp-6b7abf6b-7  proto kernel  scope link  src 169.254.127.96 

iptables from qrouter namespace in attachment

I hope that I missed something that should explain this OVS behavior, but at this time a can’t find any difference in OVS configuration when it works and when it not - and it looks like a bug.

Sergey Yezhkov


-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ovs-ofctl-dump-flows.txt
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20191203/dd3e9b78/attachment-0003.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ovs-vsctl-show.txt
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20191203/dd3e9b78/attachment-0004.txt>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: conf.db
Type: application/octet-stream
Size: 121360 bytes
Desc: not available
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20191203/dd3e9b78/attachment-0001.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: qrouter-iptables-nat.txt
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20191203/dd3e9b78/attachment-0005.txt>
-------------- next part --------------




More information about the discuss mailing list