[ovs-discuss] [ovs-dev] packet drop

Tony Liu tonyliu0592 at hotmail.com
Thu Aug 6 18:39:34 UTC 2020


Inline...

Thanks!

Tony
> -----Original Message-----
> From: Numan Siddique <numans at ovn.org>
> Sent: Thursday, August 6, 2020 10:03 AM
> To: Tony Liu <tonyliu0592 at hotmail.com>
> Cc: ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
> Subject: Re: [ovs-dev] packet drop
> 
> 
> 
> On Thu, Aug 6, 2020 at 4:05 AM Tony Liu <tonyliu0592 at hotmail.com
> <mailto:tonyliu0592 at hotmail.com> > wrote:
> 
> 
> 
> 	The drop is caused by flow change.
> 
> 	When packet is dropped.
> 	========
> 	recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,ge
> neve({class=0x102,type=0x80,len=4,0x20003/0x7fffffff}),flags(-
> df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:00:00
> /01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type=8/0
> xf8), packets:14, bytes:1372, used:0.846s, actions:drop
> 	recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df:
> 1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10.
> 6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no
> <http://192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/0x3,
> ttl=64,frag=no> ),icmp(type=0), packets:6, bytes:588, used:8.983s,
> actions:drop
> 	========
> 
> 	When packet goes through.
> 	========
> 	recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,ge
> neve({class=0x102,type=0x80,len=4,0x20003/0x7fffffff}),flags(-
> df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:00:00
> /01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type=8/0
> xf8), packets:3, bytes:294, used:0.104s, actions:12
> 	recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df:
> 1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10.
> 6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no
> <http://192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/0x3,
> ttl=64,frag=no> ),icmp(type=0), packets:3, bytes:294, used:0.103s,
> actions:ct_clear,set(tunnel(tun_id=0x1a8ee,dst=10.6.30.92,ttl=64,tp_dst=
> 6081,geneve({class=0x102,type=0x80,len=4,0x1000b}),flags(df|csum|key))),
> set(eth(src=fa:16:3e:75:b7:e5,dst=52:54:00:0c:ef:b9)),set(ipv4(ttl=63)),
> 3
> 	========
> 
> 	Is that flow programmed by ovn-controller via ovs-vswitchd?
> 
> What version of OVN and OVS are you using ?

ovn-20.03.0-2.el8.x86_64
openvswitch-2.12.0-1.el8.x86_64

> Can you share your OVN NB DB ?

Yes, I can. Let me know how.

> If I understand correctly the packet is received from the patch port to
> br-int on the gateway node and then tunnelled to the compute node right ?
> And the packet is dropped on the compute node ?

Yes and yes.

> If you could share your NB DB if it's fine with you and tell the
> destination logical port, I can try it out locally.

Here is what I had.
On compute node, ovn-controller is very busy. It keeps saying
"commit failed".
====================
2020-08-05T02:44:23.927Z|04125|reconnect|INFO|tcp:10.6.20.84:6642: connected
2020-08-05T02:44:23.936Z|04126|main|INFO|OVNSB commit failed, force recompute next time.
2020-08-05T02:44:23.938Z|04127|ovsdb_idl|INFO|tcp:10.6.20.84:6642: clustered database server is disconnected from cluster; trying another server
2020-08-05T02:44:23.939Z|04128|reconnect|INFO|tcp:10.6.20.84:6642: connection attempt timed out
2020-08-05T02:44:23.939Z|04129|reconnect|INFO|tcp:10.6.20.84:6642: waiting 2 seconds before reconnect
====================

The connection to local OVSDB keeps being dropped, because no probe
response.
========
2020-08-05T02:47:15.437Z|04351|poll_loop|INFO|wakeup due to [POLLIN] on fd 20 (10.6.20.22:42362<->10.6.20.86:6642) at lib/stream-fd.c:157 (100% CPU usage)
2020-08-05T02:47:15.438Z|04352|reconnect|WARN|tcp:127.0.0.1:6640: connection dropped (Broken pipe)
2020-08-05T02:47:15.438Z|04353|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connecting...
2020-08-05T02:47:15.449Z|04354|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt: connected
========

After set probe interval to 30s, the problem seems gone. I mentioned
this in another thread "[OVN] no response to inactivity probe".

I can restore probe interval back to default 5s, see if the problem
can be reproduced. For me, it's important to understand what happens
behind it.

> 
> Thanks
> Numan
> 
> 
> 
> 
> 
> 	Thanks!
> 
> 	Tony
> 
> 	> -----Original Message-----
> 	> From: discuss <ovs-discuss-bounces at openvswitch.org <mailto:ovs-
> discuss-bounces at openvswitch.org> > On Behalf Of Tony
> 	> Liu
> 	> Sent: Wednesday, August 5, 2020 2:48 PM
> 	> To: ovs-discuss at openvswitch.org <mailto:ovs-
> discuss at openvswitch.org> ; ovs-dev at openvswitch.org <mailto:ovs-
> dev at openvswitch.org>
> 	> Subject: [ovs-discuss] packet drop
> 	>
> 	> Hi,
> 	>
> 	> I am running ping from external to VM via OVN gateway.
> 	> On the compute node, ICMP request packet is consistently coming
> into
> 	> interface "ovn-gatewa-1". But there is about 10 out of 25 packet
> loss on
> 	> tap interface. It's like the switch pauses 10s after every 15s.
> 	>
> 	> Has anyone experiences such issue?
> 	> Any advice how to look into it?
> 	>
> 	> ========
> 	> 21fed09f-909e-4efc-b117-f5d5fcb636c9
> 	>     Bridge br-int
> 	>         fail_mode: secure
> 	>         datapath_type: system
> 	>         Port "ovn-gatewa-0"
> 	>             Interface "ovn-gatewa-0"
> 	>                 type: geneve
> 	>                 options: {csum="true", key=flow,
> remote_ip="10.6.30.91"}
> 	>                 bfd_status: {diagnostic="No Diagnostic",
> flap_count="1",
> 	> forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up,
> 	> state=up}
> 	>         Port "tap2588bb4e-35"
> 	>             Interface "tap2588bb4e-35"
> 	>         Port "ovn-gatewa-1"
> 	>             Interface "ovn-gatewa-1"
> 	>                 type: geneve
> 	>                 options: {csum="true", key=flow,
> remote_ip="10.6.30.92"}
> 	>                 bfd_status: {diagnostic="No Diagnostic",
> flap_count="1",
> 	> forwarding="true", remote_diagnostic="No Diagnostic",
> remote_state=up,
> 	> state=up}
> 	>         Port "tap37f6b2d7-cc"
> 	>             Interface "tap37f6b2d7-cc"
> 	>         Port "tap2c4b3b0f-8b"
> 	>             Interface "tap2c4b3b0f-8b"
> 	>         Port "tap23245491-a4"
> 	>             Interface "tap23245491-a4"
> 	>         Port "tap51660269-2c"
> 	>             Interface "tap51660269-2c"
> 	>         Port "tap276cd1ef-e1"
> 	>             Interface "tap276cd1ef-e1"
> 	>         Port "tap138526d3-b3"
> 	>             Interface "tap138526d3-b3"
> 	>         Port "tapd1ae48a1-2d"
> 	>             Interface "tapd1ae48a1-2d"
> 	>         Port br-int
> 	>             Interface br-int
> 	>                 type: internal
> 	>         Port "tapdd08f476-94"
> 	>             Interface "tapdd08f476-94"
> 	> ========
> 	>
> 	>
> 	> Thanks!
> 	>
> 	> Tony
> 	>
> 	> _______________________________________________
> 	> discuss mailing list
> 	> discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> 	> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> 	_______________________________________________
> 	dev mailing list
> 	dev at openvswitch.org <mailto:dev at openvswitch.org>
> 	https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 
> 



More information about the discuss mailing list