[ovs-discuss] OVS VXLAN local_ip does not seem to function - Bug
Padgett, Marcus
Marcus.Padgett at windstream.com
Thu May 24 12:18:03 UTC 2018
Haha, you're not wrong, definitely a bit of a unique scenario, but we have a specific use-case we're exploring.
This one is back to running 2.5.4 as I did not see any change in symptoms up to 2.9.1. Also, because 2.5.4 is labeled as the current LTS release (although I do see 2.5.5 was just made available this week).
I've added the requested outputs as well consolidated some others to hopefully help paint the full picture in one grouping. Please let me know if there are any other questions.
ovs-vsctl show;
38955fc8-079e-46bc-a27b-633c192f795b
Bridge "wan1"
Port "wan1"
Interface "wan1"
type: internal
Port "ens3"
Interface "ens3"
Bridge br-int
Controller "tcp:40.138.35.150:6653"
is_connected: true
Port "test-tun-2"
Interface "test-tun-2"
type: vxlan
options: {key="102", local_ip="198.15.1.20", remote_ip="40.138.35.166"}
Port "test-tun-1"
Interface "test-tun-1"
type: vxlan
options: {key="101", local_ip="198.15.0.22", remote_ip="40.138.35.166"}
Port br-int
Interface br-int
type: internal
Bridge "wan2"
Port "wan2"
Interface "wan2"
type: internal
Port "ens4"
Interface "ens4"
ovs_version: "2.5.4"
ip addr show;
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
link/ether fa:16:3e:68:f2:db brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe68:f2db/64 scope link
valid_lft forever preferred_lft forever
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc pfifo_fast master ovs-system state UP group default qlen 1000
link/ether fa:16:3e:00:72:72 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe00:7272/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:af:bc:d4:8f brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 scope global docker0
valid_lft forever preferred_lft forever
5: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1
link/ether 1e:ff:6b:19:59:c1 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1
link/ether 7a:e4:bf:b8:75:40 brd ff:ff:ff:ff:ff:ff
inet6 fe80::78e4:bfff:feb8:7540/64 scope link
valid_lft forever preferred_lft forever
10: wan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1
link/ether fa:16:3e:68:f2:db brd ff:ff:ff:ff:ff:ff
inet 198.15.0.22/24 brd 198.15.0.255 scope global wan1
valid_lft forever preferred_lft forever
inet6 fe80::8065:7aff:fe92:e543/64 scope link
valid_lft forever preferred_lft forever
11: wan2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1
link/ether fa:16:3e:00:72:72 brd ff:ff:ff:ff:ff:ff
inet 198.15.1.20/24 brd 198.15.1.255 scope global wan2
valid_lft forever preferred_lft forever
inet6 fe80::3c2b:eff:fecb:1047/64 scope link
valid_lft forever preferred_lft forever
12: vxlan_sys_4789: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65485 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether d2:59:ff:d9:7c:af brd ff:ff:ff:ff:ff:ff
inet6 fe80::d059:ffff:fed9:7caf/64 scope link
valid_lft forever preferred_lft forever
ip route show;
default via 198.15.0.1 dev wan1 metric 10
default via 198.15.1.1 dev wan2 metric 40
169.254.169.254 via 198.15.0.1 dev wan1 metric 10
169.254.169.254 via 198.15.1.1 dev wan2 metric 40
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
198.15.0.0/24 dev ens3 proto kernel scope link src 198.15.0.22
198.15.0.0/24 dev wan1 proto kernel scope link src 198.15.0.22
198.15.1.0/24 dev wan2 proto kernel scope link src 198.15.1.20
ip route show table wan1;
default via 198.15.0.1 dev wan1 src 198.15.0.22
ip route show table wan2;
default via 198.15.1.1 dev wan2 src 198.15.1.20
iptables -L -n -t mangle;
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s 198.15.0.22 -j MARK --set-mark 0xa1
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s 198.15.1.20 -j MARK --set-mark 0xa2
ip rule show;
0: from all fwmark 0xa1 lookup wan1
0: from all fwmark 0xa2 lookup wan2
10: from all lookup local
32766: from all lookup main
32767: from all lookup default
Here's a diagram if it helps as well;
[cid:image002.png at 01D3F337.0575E680]
All interfaces are behind 1-to-1 NAT, and are translated as such;
WAN1 = 198.15.0.1 = 40.138.35.146
WAN2 = 198.15.1.1 = 40.138.35.155
Data Center side is the 40.138.35.166
Thank you,
Marcus Padgett
Sr Engineer - Service Architecture | Windstream
From: Gregory Rose [mailto:gvrose8192 at gmail.com]
Sent: Wednesday, May 23, 2018 7:06 PM
To: Padgett, Marcus <Marcus.Padgett at windstream.com>; bugs at openvswitch.org
Subject: Re: [ovs-discuss] OVS VXLAN local_ip does not seem to function - Bug
On 5/21/2018 4:42 AM, Padgett, Marcus wrote:
My responses are inline.
Thank you,
Marcus Padgett
Sr Engineer - Service Architecture | Windstream
From: Gregory Rose [mailto:gvrose8192 at gmail.com]
Sent: Friday, May 18, 2018 5:55 PM
To: Padgett, Marcus <Marcus.Padgett at windstream.com><mailto:Marcus.Padgett at windstream.com>; bugs at openvswitch.org<mailto:bugs at openvswitch.org>
Subject: Re: [ovs-discuss] OVS VXLAN local_ip does not seem to function - Bug
Hi Marcus,
I have a couple questions inline below...
On 4/26/2018 6:17 AM, Padgett, Marcus wrote:
I have an issue with OVS VXLAN and using the local_ip option for the interface. Some of the IP addresses have been changed in the email since I don't know exactly who sees this.
What you did that make the problem appear.
Using a server with two interfaces with two default routes, I want to build a VXLAN tunnel over each link to a destination switch.
They both go the same switch? Is it a real switch or a SW switch like OVS? [MP] - They do both go to the same switch (single interface). Could be either hardware or SW switch long term, but right now I'm terminating them to another OVS bridge.
I am utilizing iptables to mark and ip rules to re-direct the traffic for the second tunnel out the correct interface. I have built the VXLAN tunnel in OVS trying to use the "local_ip" option.
Please provide the iptables and ip rules you used to redirect the traffic. [MP] - Iptables and ip rules below;
ip route add default via $WAN1_GW dev wan1 src $WAN1_IP table wan1
ip route add default via $WAN2_GW dev wan2 src $WAN2_IP table wan2
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s $WAN1_IP -j MARK --set-mark 0xa1
iptables -t mangle -A OUTPUT -p udp --dport 4789 -s $WAN2_IP -j MARK --set-mark 0xa2
ip rule add table local priority 10
ip rule del table local priority 0
ip rule add fwmark 0xa1 table wan1 priority 0
ip rule add fwmark 0xa2 table wan2 priority 0
So, I'm trying to capture the packet as it's generated by the process and add a MARK, prior to any routing decisions being made (OUTPUT chain in MANGLE table). What I'm seeing (and my assumption) is that the VXLAN packet is generating its source IP address based on the local or main routing table while still being handled by some OVS process. I can use the same setup with GRE (change the protocol type and remote destination port) and it works as expected.
Wow. OK, it's quite the setup and not something simple to configure and get going in an hour or two. I'm working on it... but let me ask something else.
What is the output of 'ovs-vsctl show' and 'ip addr show' and 'ip route show'? Please substitute the IP addresses shown with the $WAN1_IP and $WAN1_GW strings so that I can associate interfaces to addresses without you having to publish the actual IP address values.
Thanks,
- Greg
I set something up similar to this situation and didn't see a problem but I wasn't using a real switch.
Also, did you ever have a chance to try out a new version of OVS? As Ben mentioned 2.5.2 is pretty old. [MP] - Yes. I was able to try this in 2.5.2, 2.5.4, 2.6.1, and 2.9.1. All seemed to have the same issue.
Thanks,
- Greg
EXAMPLE:
sudo ovs-vsctl add-port ovs-br1 tun1 -- set Interface tun1 type=vxlan options:remote_ip=172.17.253.1 options:key=testflow2 options:local_ip=172.16.253.16
What you expected to happen.
I expected the VXLAN tunnel to be sent using the specified source IP address of 172.16.253.16 out the correct interface.
What actually happened.
The VXLAN tunnel exits the correct interface based on my routing rules, however was formed utilizing the other interface's IP address (172.16.252.108).
EXAMPLE:
08:24:42.779236 IP 172.16.252.108.57418 > 172.17.253.1.4789: VXLAN, flags [I] (0x08), vni 0
LLDP, length 79
08:24:42.779437 IP 172.16.252.108.34566 > 172.17.253.1.4789: VXLAN, flags [I] (0x08), vni 0
02:eb:86:0d:38:74 (oui Unknown) > Broadcast, ethertype Unknown (0x8942), length 93:
0x0000: 0207 0486 e5fb d3da 4204 0502 0000 000e ........B.......
0x0010: 0602 0078 fe12 a423 0501 4f4e 4f53 2044 ...x...#..ONOS.D
0x0020: 6973 636f 7665 7279 fe17 a423 0502 6f66 iscovery...#..of
0x0030: 3a30 3030 3038 3665 3566 6264 3364 6134 :000086e5fbd3da4
0x0040: 3208 0a75 6370 652d 7475 6e2d 3200 00 2..tun1..
The Open vSwitch version number (as output by ovs-vswitchd --version).
ovs-vswitchd (Open vSwitch) 2.5.2
Compiled Oct 17 2017 16:38:57
The kernel version on which Open vSwitch is running (from /proc/version) and the distribution and version number of your OS (e.g. "Centos 5.0").
Linux version 4.4.0-87-generic (buildd at lcy01-31) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.4) ) #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017
The output of ovs-dpctl show.
system at ovs-system:
lookups: hit:191881 missed:5814 lost:0
flows: 4
masks: hit:471168 total:4 hit/pkt:2.38
port 0: ovs-system (internal)
port 1: ovs-br1 (internal)
port 2: vxlan_sys_4789 (vxlan)
port 3: ovs-lan (internal)
port 4: k8s-br (internal)
port 5: mirror-br (internal)
port 6: ens6
port 7: wan2 (internal)
Any other information that you think might be relevant.
Everything works fine when building with GRE instead of VXLAN, all the same routing and firewall rules. The rules are not matching any protocol specific parameters, just matching on destination IP address to mark the traffic.
Thank you,
Marcus Padgett
Sr Engineer - Service Architecture | Windstream
Marcus.Padgett at windstream.com<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.windstreambusiness.com%2F&data=02%7C01%7CMarcus.Padgett%40windstream.com%7C32c54daf46044a07ab7008d5bd0a2310%7C2567b4c1b0ed40f5aee358d7c5f3e2b2%7C1%7C0%7C636622773597849997&sdata=%2BHevGor9WR8uzFbSuS%2FuALFmZM74B6nZ91u2Fl%2F%2Bd3k%3D&reserved=0>
This email message and any attachments are for the sole use of the intended recipient(s). Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message and any attachments.
_______________________________________________
discuss mailing list
discuss at openvswitch.org<mailto:discuss at openvswitch.org>
https://mail.openvswitch.org/mailman/listinfo/ovs-discuss<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.openvswitch.org%2Fmailman%2Flistinfo%2Fovs-discuss&data=02%7C01%7CMarcus.Padgett%40windstream.com%7C32c54daf46044a07ab7008d5bd0a2310%7C2567b4c1b0ed40f5aee358d7c5f3e2b2%7C1%7C0%7C636622773597860005&sdata=WsT%2BIoPQIOjKD%2FdjtKmyh3YKe7jqsuY6AqP4zl9J2G0%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20180524/5419618b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 56859 bytes
Desc: image002.png
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20180524/5419618b/attachment-0001.png>
More information about the discuss
mailing list