[ovs-discuss] Open-v-switch flow issue.

Numan Siddique numans at ovn.org
Mon Mar 16 11:27:11 UTC 2020


On Wed, Mar 11, 2020 at 2:08 AM Vadim M Basyrov <VBasyrov at technoserv.com> wrote:
>
> Hello!
>
> I write to you because I can’t solve one issue regarding open-v-switch.
>
> We have the OpenStack deployment – version Queens.
>

In the case of openstack, neutron ovs agent programs the flows right ?
I think you should investigate whether neutron ovs agent is
programming the required flows or not.

Thanks
Numan

> 11 nodes – 3 control nodes and 8 compute nodes.
>
> There are seven compute nodes with ovs-vswitchd (Open vSwitch) 2.9.0 and last one compute node with ovs-vswitchd (Open vSwitch) 2.9.5 (from the start we had ovs-vswitchd (Open vSwitch) 2.9.0 like as on other nodes).
>
> We upgraded ovs-vswitchd (Open vSwitch) 2.9.5 on the last compute node for test and wanted to figure out that this problem will be persists or not.
>
>
> When we launch virtual machine on first seven nodes, virtual machine works and gets ip-addresses from dhcp without problem.
>
> But when we launch virtual machine or migrate virtual machine on the last compute node virtual machine does not get ip-addresses from dhcp.
>
>
> We investigated this problem and found one interesting moment.
>
> On the first seven nodes after creating virtual machine we can see correct dump flow for private network of virtual machine.
>
>
> Correct dump:
>
> root at dc2-compute03:~# ovs-ofctl dump-flows br-tun | grep 0x10057
>
> cookie=0x7e0b101c14974b46, duration=8712088.227s, table=4, n_packets=58210459, n_bytes=333529723572, idle_age=39, hard_age=65534, priority=1,tun_id=0x10057 actions=mod_vlan_vid:3,resubmit(,9)
>
>  cookie=0x7e0b101c14974b46, duration=2305838.428s, table=20, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,dl_vlan=3,dl_dst=fa:16:3e:f7:e1:46 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:6
>
>  cookie=0x7e0b101c14974b46, duration=2305820.317s, table=20, n_packets=54, n_bytes=6045, idle_age=65534, hard_age=65534, priority=2,dl_vlan=3,dl_dst=fa:16:3e:10:65:23 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:3
>
>  cookie=0x7e0b101c14974b46, duration=2305811.544s, table=20, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,dl_vlan=3,dl_dst=fa:16:3e:2c:92:0d actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:10
>
>  cookie=0x7e0b101c14974b46, duration=2305795.834s, table=20, n_packets=45, n_bytes=4175, idle_age=65534, hard_age=65534, priority=2,dl_vlan=3,dl_dst=fa:16:3e:ea:d7:e4 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:4
>
>  cookie=0x7e0b101c14974b46, duration=2305795.238s, table=20, n_packets=36, n_bytes=1865, idle_age=65534, hard_age=65534, priority=2,dl_vlan=3,dl_dst=fa:16:3e:65:23:5d actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:5
>
>  cookie=0x7e0b101c14974b46, duration=1027284.390s, table=20, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2,dl_vlan=3,dl_dst=fa:16:3e:90:b8:73 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:8
>
>  cookie=0x7e0b101c14974b46, duration=1027268.690s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=65534, hard_age=38, priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:90:b8:73 actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:8
>
>  cookie=0x7e0b101c14974b46, duration=23829.098s, table=20, n_packets=220, n_bytes=14712, hard_timeout=300, idle_age=166, hard_age=124, priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:b6:4d:f1 actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:3
>
>  cookie=0x7e0b101c14974b46, duration=131.880s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=131, priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:10:65:23 actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:3
>
>  cookie=0x7e0b101c14974b46, duration=131.880s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=131, priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:65:23:5d actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:5
>
>  cookie=0x7e0b101c14974b46, duration=131.880s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=131, priority=1,vlan_tci=0x0003/0x0fff,dl_dst=fa:16:3e:ea:d7:e4 actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:4
>
>  cookie=0x7e0b101c14974b46, duration=2305817.320s, table=22, n_packets=27156, n_bytes=3159361, idle_age=24888, hard_age=65534, priority=1,dl_vlan=3 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:3,output:4,output:5,output:6,output:8,output:10
>
>
> On the last compute node after creating virtual machine we have incorrect dump flow or incomplete. In this case we don’t see any “NXM_OF_VLAN_TCI” output strings to MAC-addresses of DHCP.
>
>
> Incorrect dump:
>
> root at dc2-compute09:~# ovs-ofctl dump-flows br-tun | grep 0x10057
>
>  cookie=0x189ebcbc0bc4a360, duration=3006.464s, table=4, n_packets=124, n_bytes=24219, idle_age=21, priority=1,tun_id=0x10057 actions=mod_vlan_vid:6,resubmit(,9)
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.196s, table=20, n_packets=0, n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:65:23:5d actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:22
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.189s, table=20, n_packets=0, n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:10:65:23 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:23
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.183s, table=20, n_packets=0, n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:ea:d7:e4 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:24
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.175s, table=20, n_packets=0, n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:0f:49:0c actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:26
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.169s, table=20, n_packets=0, n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:f7:e1:46 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:27
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.160s, table=20, n_packets=0, n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:90:b8:73 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:29
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.152s, table=20, n_packets=0, n_bytes=0, idle_age=3005, priority=2,dl_vlan=6,dl_dst=fa:16:3e:2c:92:0d actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:31
>
>  cookie=0x189ebcbc0bc4a360, duration=2904.220s, table=20, n_packets=0, n_bytes=0, hard_timeout=300, idle_age=2904, hard_age=21, priority=1,vlan_tci=0x0006/0x0fff,dl_dst=fa:16:3e:90:b8:73 actions=load:0->NXM_OF_VLAN_TCI[],load:0x10057->NXM_NX_TUN_ID[],output:29
>
>  cookie=0x189ebcbc0bc4a360, duration=3003.156s, table=22, n_packets=63, n_bytes=16788, idle_age=3, priority=1,dl_vlan=6 actions=strip_vlan,load:0x10057->NXM_NX_TUN_ID[],output:22,output:23,output:24,output:25,output:26,output:27,output:28,output:29,output:30,output:31
>
>
> OS on all hosts:
>
> NAME="Ubuntu"
>
> VERSION="16.04.4 LTS (Xenial Xerus)"
>
> ID=ubuntu
>
> ID_LIKE=debian
>
> PRETTY_NAME="Ubuntu 16.04.4 LTS"
>
> VERSION_ID="16.04"
>
> HOME_URL="http://www.ubuntu.com/"
>
> SUPPORT_URL="http://help.ubuntu.com/"
>
> BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
>
> VERSION_CODENAME=xenial
>
> UBUNTU_CODENAME=xenial
>
>
> Kernel version on the first seven nodes: 4.4.0-130-lowlatency
>
> Kernel version on the last node: 4.4.0-170-generic (before was 4.4.0-130-lowlatency when we found problem, we upgraded open-v-switch version and kernel version)
>
>
> I've attached two conf.db files of correct and incorrect node. Correct node where network works well and incorrect node where network does not work well (the last compute node).
>
>
> "ovs-dpctl show" command dump of correct node.
>
> root at dc2-compute03:~# ovs-dpctl show
>
> system at ovs-system:
>
>         lookups: hit:304985044225 missed:261957039 lost:108
>
>         flows: 918
>
>         masks: hit:4793394128544 total:17 hit/pkt:15.70
>
>         port 0: ovs-system (internal)
>
>         port 1: vxlan_sys_4789 (vxlan: packet_type=ptap)
>
>         port 2: br-tun (internal)
>
>         port 3: vlan944 (internal)
>
>         port 4: vlan941 (internal)
>
>         port 5: vlan940 (internal)
>
>         port 6: qvo89695e65-21
>
>         port 7: qvoc1d94fa7-f3
>
>         port 8: qvofd9e30ab-8b
>
>         port 9: qr-4600256d-b3 (internal)
>
>         port 10: qvo53120329-cc
>
>         port 11: qr-36735708-b9 (internal)
>
>         port 12: qr-2dcb6c75-e5 (internal)
>
>         port 13: qr-de554b3e-ee (internal)
>
>         port 14: qr-132fbe52-82 (internal)
>
>         port 15: qr-3b3d0596-39 (internal)
>
>         port 16: qvo21070b6b-44
>
>         port 17: qr-11fb5e85-16 (internal)
>
>         port 18: qr-1d58adb4-45 (internal)
>
>         port 19: qr-5a3db12a-f9 (internal)
>
>         port 20: br-int (internal)
>
>         port 21: qr-cfe0668e-2f (internal)
>
>         port 22: qr-e04a70fc-ff (internal)
>
>         port 23: qr-9f0848e5-e7 (internal)
>
>         port 24: qr-6a9dec56-3f (internal)
>
>         port 25: br0 (internal)
>
>         port 26: enp59s0f0
>
>         port 27: enp216s0f1
>
>         port 28: vlan123 (internal)
>
>         port 29: vlan943 (internal)
>
>         port 30: qr-fc0718b4-e8 (internal)
>
>         port 31: qr-ed5db182-b6 (internal)
>
>         port 32: qvob6b62461-6c
>
>         port 33: qvod2d02a50-13
>
>         port 36: qr-2fdc16d7-a3 (internal)
>
>         port 39: qvoed4060b4-ed
>
>         port 42: qvob6dd5c36-c5
>
>         port 43: qr-c091f908-ba (internal)
>
>         port 45: fg-d63f59c5-8f (internal)
>
>         port 46: qvo478e4755-8d
>
>         port 47: qr-72a3e62b-c2 (internal)
>
>         port 48: qvo7c88a4c8-31
>
>         port 49: qvoc8505821-7a
>
>         port 50: qvo374ab104-81
>
>         port 52: qvode477341-12
>
>         port 53: qvo74fa0739-24
>
>         port 55: qvof415d31d-0e
>
>         port 56: qvob727e682-40
>
>         port 57: qvo2676d8ec-53
>
>         port 58: qvoa87b5c32-85
>
>         port 59: qr-785b0b3f-78 (internal)
>
>         port 60: qvo1192309a-b7
>
>         port 61: qvo736dffdf-8e
>
>         port 62: qvo673e82c7-3c
>
>         port 63: qvoe0ffcdfa-bd
>
>         port 64: qvo34b3d411-dd
>
>         port 65: qvoc032b53f-0b
>
>         port 66: qvo7a77a59e-84
>
>         port 68: qvo859183e9-30
>
>         port 69: qvo70596e32-ba
>
>         port 70: qvoebbb1420-a7
>
>         port 71: qvo800895af-c8
>
>         port 73: qvo186f5253-12
>
>         port 75: qr-a1164d85-22 (internal)
>
>         port 76: qvob1c5ab98-ac
>
>         port 79: qr-5ea9b7ad-8e (internal)
>
>         port 81: qr-d20a18c2-0c (internal)
>
>         port 84: qvo7b1ef463-1a
>
>         port 85: qr-d14d71f2-4c (internal)
>
>         port 86: qr-f6b5a89e-16 (internal)
>
>         port 88: qvo0bc1c446-d0
>
>         port 89: qr-20b48617-ce (internal)
>
>         port 98: qvo4a760e85-7e
>
>         port 99: qvofcb1f977-e9
>
>
> "ovs-dpctl show" command dump of incorrect node.
>
> root at dc2-compute09:~# ovs-dpctl show
>
> system at ovs-system:
>
>         lookups: hit:2747212618 missed:79153734 lost:0
>
>         flows: 161
>
>         masks: hit:16755259144 total:9 hit/pkt:5.93
>
>         port 0: ovs-system (internal)
>
>         port 1: vxlan_sys_4789 (vxlan: packet_type=ptap)
>
>         port 2: br-tun (internal)
>
>         port 3: br0 (internal)
>
>         port 4: enp216s0f1
>
>         port 5: enp59s0f0
>
>         port 6: bond0 (internal)
>
>         port 7: vlan123 (internal)
>
>         port 8: vlan943 (internal)
>
>         port 9: vlan944 (internal)
>
>         port 10: br-int (internal)
>
>         port 11: vlan941 (internal)
>
>         port 12: vlan940 (internal)
>
>         port 13: qvoad8dfa6f-cf
>
>         port 14: qr-1d58adb4-45 (internal)
>
>         port 15: qr-c091f908-ba (internal)
>
>         port 16: fg-d52bf3cc-aa (internal)
>
>
> Have I right found that flow hasn't TCI string to DHCP?
>
>
> Best regards,
>
> Vadim Basyyrov
>
> System engineer of
>
> Cloud solutions department
>
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


More information about the discuss mailing list