[ovs-dev] BUG: Application write syscall throttled over Open vSwitch + VxLAN

Jean Tourrilhes jean.tourrilhes at hpe.com
Fri Mar 6 22:31:52 UTC 2020


On Wed, Mar 04, 2020 at 05:13:54PM -0800, Jean Tourrilhes wrote:
> 	Hi all,
> 
> 	I'm facing a puzzling bug [...]

	One more detail I did not mention.
	The vswitchd log does not show very much. It kinds of confirm
that the packets are processed in software : I see way less upcalls in
the log than the thousands os packets being throttled. I suspect I'm
not using the right debug options.

	There is only one thing I see in the log that is kind of suspicious :
----------------------------------------------------
2020-03-05T17:09:19.611Z|00292|dpif(handler86)|DBG|system at ovs-system: failed to put[create] (File exists) ufid:235eeb07-d889-4b9a-a7bb-f3326a288b66 recirc_id(0),dp_hash(0/0),skb_priority(0/0),tunnel(tun_id=0x0,src=10.67.67.40,dst=10.67.67.41,ttl=64/0,tp_src=58801/0,tp_dst=4789/0,flags(-df-csum+key)),in_port(2),skb_mark(0/0),ct_state(0/0),ct_zone(0/0),ct_mark(0/0),ct_label(0/0),eth(src=06:a0:2a:f6:8c:4d/00:00:00:00:00:00,dst=fe:15:17:a3:0c:47/00:00:00:00:00:00),eth_type(0x0800),ipv4(src=10.68.58.40/0.0.0.0,dst=10.67.57.41/0.0.0.0,proto=17,tos=0/0,ttl=64/0,frag=no),udp(src=138/0,dst=138/0x800), actions:1
----------------------------------------------------
	This is repeated many times.

	This is the state of the flow table after it throttle my app :
---------------------------------------------------------------
recirc_id(0),tunnel(tun_id=0x0,src=10.65.65.40,dst=10.65.65.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:97, bytes:6402, used:0.300s, actions:userspace(pid=2175513357,slow_path(bfd))
recirc_id(0),tunnel(tun_id=0x0,src=10.67.67.40,dst=10.67.67.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:99, bytes:6534, used:0.040s, actions:userspace(pid=2175513357,slow_path(bfd))
recirc_id(0),in_port(1),eth(),eth_type(0x0800),ipv4(dst=10.68.58.0/255.255.255.0,proto=6,tos=0/0x3,frag=no), packets:432, bytes:45136, used:0.084s, flags:SFP., actions:set(tunnel(tun_id=0x0,dst=10.68.68.40,ttl=64,tp_dst=4789,flags(df|key))),2
recirc_id(0),tunnel(tun_id=0x0,src=10.62.62.40,dst=10.62.62.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:97, bytes:6402, used:0.044s, actions:userspace(pid=2175513357,slow_path(bfd))
recirc_id(0),tunnel(tun_id=0x0,src=10.68.68.40,dst=10.68.68.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=6,frag=no), packets:553, bytes:74098, used:0.084s, flags:SFP., actions:1
recirc_id(0),tunnel(tun_id=0x0,src=10.64.64.40,dst=10.64.64.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:98, bytes:6468, used:0.328s, actions:userspace(pid=2175513357,slow_path(bfd))
recirc_id(0),tunnel(tun_id=0x0,src=10.66.66.40,dst=10.66.66.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:98, bytes:6468, used:0.080s, actions:userspace(pid=2175513357,slow_path(bfd))
recirc_id(0),tunnel(tun_id=0x0,src=10.68.68.40,dst=10.68.68.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=4096/0x1000), packets:120, bytes:8800, used:0.576s, actions:1
recirc_id(0),in_port(1),eth(),eth_type(0x0800),ipv4(dst=10.68.58.0/255.255.255.0,proto=17,tos=0/0x3,frag=no),udp(dst=0/0x800), packets:10016, bytes:14419175, used:6.528s, actions:set(tunnel(tun_id=0x0,dst=10.68.68.40,ttl=64,tp_dst=4789,flags(df|key))),2
recirc_id(0),tunnel(tun_id=0x0,src=10.63.63.40,dst=10.63.63.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:97, bytes:6402, used:0.640s, actions:userspace(pid=2175513357,slow_path(bfd))
recirc_id(0),tunnel(tun_id=0x0,src=10.68.68.40,dst=10.68.68.41,flags(-df-csum+key)),in_port(2),eth(),eth_type(0x0800),ipv4(proto=17,frag=no),udp(dst=3784), packets:251, bytes:16566, used:0.620s, actions:userspace(pid=2175513357,slow_path(bfd))
---------------------------------------------------------------

	I wondering if the weird masking on the UDP transport port is
causing the issue. My flow rules wildcard the transport ports, so I've
no idea where it's comming from.
	Thanks in advance !

	Regards,

	Jean


More information about the dev mailing list