[ovs-discuss] VXLAN and ARP issues

Christian Colic c at ccolic.ch
Tue Dec 4 08:46:07 UTC 2018


Hi there,

I am trying to use OVS to extend a central layer 2 network to any
clients that connect to it with VXLAN.

My setup is as follows:


central01 - centos7, ovs-vsctl (Open vSwitch) 2.10.1
This virtual system has two NICs in the same VLAN (450). There is no
dot1q tagging going on, just untagged access ports.

This is how I have set up the ovs bridges:

[root at xyz ~]# ovs-vsctl show
69c043f8-a0c9-4fae-905e-2804cf3dc776
    Bridge br-internal
        Port "vx0"
            Interface "vx0"
                type: vxlan
                options: {remote_ip="169.254.0.11"}
        Port "eth1"
            Interface "eth1"
        Port br-internal
            Interface br-internal
                type: internal
    Bridge br-wan    <--- 193.5.80.179/24
        Port "eth0"
            Interface "eth0"
        Port br-wan
            Interface br-wan
                type: internal
    ovs_version: "2.10.1"

Outside of this, I have a Wireguard interface wg0, which connects to
client01, which has IP 169.254.0.1

I am able to ping the client on this IP and to me it looks like the
VXLAN tunnel is correctly being connected.

I tried this setup with only one NIC in VLAN450, but I had the same
issues. Now I tried it with two, just to be able to separate the
interfaces to 2 different bridges.

ip address output for central01:

[root at xyz ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
master ovs-system state UP group default qlen 1000
    link/ether 00:50:56:bd:9d:c3 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::250:56ff:febd:9dc3/64 scope link
       valid_lft forever preferred_lft forever
15: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state
UNKNOWN group default qlen 1000
    link/none
    inet 169.254.0.1/23 scope global wg0
       valid_lft forever preferred_lft forever
    inet6 fe80::f132:a317:8cda:c16b/64 scope link flags 800
       valid_lft forever preferred_lft forever
19: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq
master ovs-system state UP group default qlen 1000
    link/ether 00:50:56:bd:bd:4d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::250:56ff:febd:bd4d/64 scope link
       valid_lft forever preferred_lft forever
24: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
group default qlen 1000
    link/ether f6:c6:e0:bc:db:5f brd ff:ff:ff:ff:ff:ff
25: br-internal: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500
qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 00:50:56:bd:bd:4d brd ff:ff:ff:ff:ff:ff
    inet6 fde2:32a6:6a01:ff00:250:56ff:febd:bd4d/64 scope global
mngtmpaddr dynamic
       valid_lft 2591956sec preferred_lft 604756sec
    inet6 2001:620:500:fff0:250:56ff:febd:bd4d/64 scope global
mngtmpaddr dynamic
       valid_lft 2591956sec preferred_lft 604756sec
    inet6 2001:620:500:ff00:250:56ff:febd:bd4d/64 scope global
mngtmpaddr dynamic
       valid_lft 2591956sec preferred_lft 604756sec
    inet6 fe80::6487:3eff:fe9b:3640/64 scope link
       valid_lft forever preferred_lft forever
26: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UNKNOWN group default qlen 1000
    link/ether 00:50:56:bd:9d:c3 brd ff:ff:ff:ff:ff:ff
    inet 193.5.80.179/24 brd 193.5.80.255 scope global br-wan
       valid_lft forever preferred_lft forever
    inet6 fde2:32a6:6a01:ff00:250:56ff:febd:9dc3/64 scope global
mngtmpaddr dynamic
       valid_lft 2591956sec preferred_lft 604756sec
    inet6 2001:620:500:fff0:250:56ff:febd:9dc3/64 scope global
mngtmpaddr dynamic
       valid_lft 2591956sec preferred_lft 604756sec
    inet6 2001:620:500:ff00:250:56ff:febd:9dc3/64 scope global
mngtmpaddr dynamic
       valid_lft 2591956sec preferred_lft 604756sec
    inet6 fe80::94b3:b7ff:feb5:c841/64 scope link
       valid_lft forever preferred_lft forever
28: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc
noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 52:e7:42:9c:41:d5 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::50e7:42ff:fe9c:41d5/64 scope link
       valid_lft forever preferred_lft forever


Now the client that is connecting:

client01, centos7, ovs-vsctl (Open vSwitch) 2.5.6

[root at centos-s-1vcpu-1gb-fra1-01 ~]# ovs-vsctl show
d2b434f3-e1c7-438a-8e44-73313ae9cef5
    Bridge br-vxlan
        Port "vx0"
            Interface "vx0"
                type: vxlan
                options: {remote_ip="169.254.0.1"}
        Port br-vxlan
            Interface br-vxlan
                type: internal
    Bridge br-wan    <--- 142.93.111.193/20
        Port br-wan
            Interface br-wan
                type: internal
        Port "eth0"
            Interface "eth0"
    ovs_version: "2.5.6"

Again outside from all of this is a Wireguard interface wg0, which
connects to the central server, and has IP 169.254.0.11

ip address output for client01:

[root at centos-s-1vcpu-1gb-fra1-01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast
master ovs-system state UP group default qlen 1000
    link/ether 9e:bc:35:33:65:be brd ff:ff:ff:ff:ff:ff
    inet6 fe80::9cbc:35ff:fe33:65be/64 scope link
       valid_lft forever preferred_lft forever
3: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN
group default qlen 1000
    link/ether 36:f4:9b:13:ff:dc brd ff:ff:ff:ff:ff:ff
18: br-vxlan: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1300 qdisc
noqueue state UNKNOWN group default qlen 1000
    link/ether f2:25:57:5c:7b:42 brd ff:ff:ff:ff:ff:ff
    inet 193.5.80.200/24 scope global br-vxlan
       valid_lft forever preferred_lft forever
    inet6 fde2:32a6:6a01:ff00:f025:57ff:fe5c:7b42/64 scope global
mngtmpaddr dynamic
       valid_lft 2591958sec preferred_lft 604758sec
    inet6 2001:620:500:fff0:f025:57ff:fe5c:7b42/64 scope global
mngtmpaddr dynamic
       valid_lft 2591958sec preferred_lft 604758sec
    inet6 2001:620:500:ff00:f025:57ff:fe5c:7b42/64 scope global
mngtmpaddr dynamic
       valid_lft 2591958sec preferred_lft 604758sec
    inet6 fe80::f025:57ff:fe5c:7b42/64 scope link
       valid_lft forever preferred_lft forever
19: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1220 qdisc noqueue
state UNKNOWN group default qlen 1000
    link/ether 9e:bc:35:33:65:be brd ff:ff:ff:ff:ff:ff
    inet 142.93.111.193/20 brd 142.93.111.255 scope global br-wan
       valid_lft forever preferred_lft forever
21: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1220 qdisc noqueue state
UNKNOWN group default qlen 1000
    link/none
    inet 169.254.0.11/23 scope global wg0
       valid_lft forever preferred_lft forever
23: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc
noqueue master ovs-system state UNKNOWN group default qlen 1000
    link/ether 42:1a:71:83:11:0a brd ff:ff:ff:ff:ff:ff
    inet6 fe80::401a:71ff:fe83:110a/64 scope link
       valid_lft forever preferred_lft forever



If I do a tcpdump on client01, interface br-vxlan I can see a lot of ARP
requests that are happening in this VLAN450, so this is correctly being
forwarded through the VXLAN tunnel:

[root at centos-s-1vcpu-1gb-fra1-01 ~]# tcpdump -i br-vxlan -vv -n
tcpdump: listening on br-vxlan, link-type EN10MB (Ethernet), capture
size 262144 bytes
08:32:07.738047 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.145 tell 193.5.80.1, length 46
08:32:07.807440 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.119 tell 193.5.80.1, length 46
08:32:07.879636 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.121 tell 193.5.80.1, length 46
08:32:07.885878 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.150 tell 193.5.80.1, length 46
08:32:07.896041 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.151 tell 193.5.80.1, length 46
08:32:07.896055 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.153 tell 193.5.80.1, length 46
[....]


Now my goal is to give the client01 an IP from this range, say
193.5.80.200/24 and be able to ping all devices in there, just in the
same way I could if I connect myself directly into the same switch
locally in VLAN450.
I tried to assign the IP to br-vxlan, but to me it looks like the ARP
requests are sometimes not forwarded correctly or are not being answered.

If I ping from central01 to 193.5.80.200, I see this on client01 br-vxlan:

[root at centos-s-1vcpu-1gb-fra1-01 ~]# tcpdump -i br-vxlan -vv -n
tcpdump: listening on br-vxlan, link-type EN10MB (Ethernet), capture
size 262144 bytes
08:44:18.048206 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.200 tell 193.5.80.179, length 46
[...]
08:44:19.050693 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has
193.5.80.200 tell 193.5.80.179, length 46
[...]

The ARP requests for his IP are coming in, but they are not being answered.

If I ping from the client to the gateway of VLAN450, I see the ICMP
request go through, but there is no reply.

The ICMP request on central01, interface eth1:

[root at xyz ~]# tcpdump -i eth1 -vv -n

09:36:45.243757 IP (tos 0x0, ttl 64, id 28527, offset 0, flags [DF],
proto ICMP (1), length 84)
    193.5.80.200 > 193.5.80.1: ICMP echo request, id 25347, seq 1, length 64

Both sides are using OVS normal mode:

[root at centos-s-1vcpu-1gb-fra1-01 ~]# ovs-ofctl -O Openflow13 dump-flows
br-vxlan
OFPST_FLOW reply (OF1.3) (xid=0x2):
 cookie=0x0, duration=1009728.779s, table=0, n_packets=517014,
n_bytes=31259810, priority=0 actions=NORMAL

I am not really sure what to test anymore. My goal in the end is to
create virtual machines on client01, connect them to br-vxlan, so that
they are directly in VLAN450, just like all local VMs in the datacenter.

I hope someone can give me some hints on what to check.

Thanks very much and best regards,
Chris



More information about the discuss mailing list