[ovs-discuss] [ovs-dev] packet drop

Tony Liu tonyliu0592 at hotmail.com
Thu Aug 6 19:21:38 UTC 2020


Inline...

Thanks!

Tony
> -----Original Message-----
> From: Numan Siddique <numans at ovn.org>
> Sent: Thursday, August 6, 2020 11:49 AM
> To: Tony Liu <tonyliu0592 at hotmail.com>
> Cc: ovs-dev at openvswitch.org; ovs-discuss at openvswitch.org
> Subject: Re: [ovs-discuss] [ovs-dev] packet drop
> 
> On Fri, Aug 7, 2020 at 12:10 AM Tony Liu <tonyliu0592 at hotmail.com> wrote:
> >
> > Inline...
> >
> > Thanks!
> >
> > Tony
> > > -----Original Message-----
> > > From: Numan Siddique <numans at ovn.org>
> > > Sent: Thursday, August 6, 2020 10:03 AM
> > > To: Tony Liu <tonyliu0592 at hotmail.com>
> > > Cc: ovs-discuss at openvswitch.org; ovs-dev at openvswitch.org
> > > Subject: Re: [ovs-dev] packet drop
> > >
> > >
> > >
> > > On Thu, Aug 6, 2020 at 4:05 AM Tony Liu <tonyliu0592 at hotmail.com
> > > <mailto:tonyliu0592 at hotmail.com> > wrote:
> > >
> > >
> > >
> > >       The drop is caused by flow change.
> > >
> > >       When packet is dropped.
> > >       ========
> > >
> > > recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,ge
> > > neve({class=0x102,type=0x80,len=4,0x20003/0x7fffffff}),flags(-
> > > df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:0
> > > df+csum+0:00
> > > /01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type
> > > =8/0 xf8), packets:14, bytes:1372, used:0.846s, actions:drop
> > >
> recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df:
> > >
> 1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10.
> > > 6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no
> > > <http://192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/
> > > 0x3, ttl=64,frag=no> ),icmp(type=0), packets:6, bytes:588,
> > > used:8.983s, actions:drop
> > >       ========
> > >
> > >       When packet goes through.
> > >       ========
> > >
> > > recirc_id(0),tunnel(tun_id=0x19aca,src=10.6.30.92,dst=10.6.30.22,ge
> > > neve({class=0x102,type=0x80,len=4,0x20003/0x7fffffff}),flags(-
> > > df+csum+key)),in_port(3),eth(src=fa:16:3e:df:1e:85,dst=00:00:00:00:0
> > > df+csum+0:00
> > > /01:00:00:00:00:00),eth_type(0x0800),ipv4(proto=1,frag=no),icmp(type
> > > =8/0 xf8), packets:3, bytes:294, used:0.104s, actions:12
> > >
> recirc_id(0),in_port(12),eth(src=fa:16:3e:7d:bb:85,dst=fa:16:3e:df:
> > >
> 1e:85),eth_type(0x0800),ipv4(src=192.168.236.152/255.255.255.252,dst=10.
> > > 6.40.9,proto=1,tos=0/0x3,ttl=64,frag=no
> > > <http://192.168.236.152/255.255.255.252,dst=10.6.40.9,proto=1,tos=0/
> > > 0x3, ttl=64,frag=no> ),icmp(type=0), packets:3, bytes:294,
> > > used:0.103s,
> > > actions:ct_clear,set(tunnel(tun_id=0x1a8ee,dst=10.6.30.92,ttl=64,tp_
> > > dst=
> > > 6081,geneve({class=0x102,type=0x80,len=4,0x1000b}),flags(df|csum|key
> > > ))),
> > > set(eth(src=fa:16:3e:75:b7:e5,dst=52:54:00:0c:ef:b9)),set(ipv4(ttl=6
> > > 3)),
> > > 3
> > >       ========
> > >
> > >       Is that flow programmed by ovn-controller via ovs-vswitchd?
> > >
> > > What version of OVN and OVS are you using ?
> >
> > ovn-20.03.0-2.el8.x86_64
> > openvswitch-2.12.0-1.el8.x86_64
> >
> > > Can you share your OVN NB DB ?
> >
> > Yes, I can. Let me know how.
> >
> > > If I understand correctly the packet is received from the patch port
> > > to br-int on the gateway node and then tunnelled to the compute node
> right ?
> > > And the packet is dropped on the compute node ?
> >
> > Yes and yes.
> >
> > > If you could share your NB DB if it's fine with you and tell the
> > > destination logical port, I can try it out locally.
> >
> > Here is what I had.
> > On compute node, ovn-controller is very busy. It keeps saying "commit
> > failed".
> > ====================
> > 2020-08-05T02:44:23.927Z|04125|reconnect|INFO|tcp:10.6.20.84:6642:
> > connected 2020-08-05T02:44:23.936Z|04126|main|INFO|OVNSB commit failed,
> force recompute next time.
> > 2020-08-05T02:44:23.938Z|04127|ovsdb_idl|INFO|tcp:10.6.20.84:6642:
> > clustered database server is disconnected from cluster; trying another
> > server
> > 2020-08-05T02:44:23.939Z|04128|reconnect|INFO|tcp:10.6.20.84:6642:
> > connection attempt timed out
> > 2020-08-05T02:44:23.939Z|04129|reconnect|INFO|tcp:10.6.20.84:6642:
> > waiting 2 seconds before reconnect ====================
> >
> > The connection to local OVSDB keeps being dropped, because no probe
> > response.
> > ========
> > 2020-08-05T02:47:15.437Z|04351|poll_loop|INFO|wakeup due to [POLLIN]
> > on fd 20 (10.6.20.22:42362<->10.6.20.86:6642) at lib/stream-fd.c:157
> > (100% CPU usage)
> > 2020-08-05T02:47:15.438Z|04352|reconnect|WARN|tcp:127.0.0.1:6640:
> > connection dropped (Broken pipe)
> > 2020-08-
> 05T02:47:15.438Z|04353|rconn|INFO|unix:/var/run/openvswitch/br-int.mgmt:
> connecting...
> > 2020-08-05T02:47:15.449Z|04354|rconn|INFO|unix:/var/run/openvswitch/br
> > -int.mgmt: connected ========
> >
> > After set probe interval to 30s, the problem seems gone. I mentioned
> > this in another thread "[OVN] no response to inactivity probe".
> >
> > I can restore probe interval back to default 5s, see if the problem
> > can be reproduced. For me, it's important to understand what happens
> > behind it.
> 
> Ok. This makes sense. Since the connection between ovn-controller and
> ovs-vswitchd is braking, the flows are deleted and reprogrammed and this
> results in a loop. And the packet gets dropped if flows are missing.

When the connection between ovn-controller and ovs-vswitchd is broken,
all flows in ovs-vswitchd are cleaned up?

I did see missing lookups from ovs-vswitchd statistics. So the flow
is cleaned on slow path, which results the drop action for flow on
fast path.

Is ovn-controller considered as part of control plane or data plane?
Because the data plane is supposed to keep running while some glitches
happen in control plane.

Is there any way to retain flows when ovn-controller is disconnected
from ovs-vswitchd?

> I think increasing the probe interval - ovn-openflow-probe-interval
> seems fine to me. It indicates that ovn-controller is taking more than
> 5 seconds to do a full recompute.

Yes, that helps. I am discussing this in another thread "[OVN] no
response to inactivity probe". Three questions there.
1. UNIX or TCP, which is recommended for ovn-controller to connect
   to ovs-vswitchd.
2. Is there probe on UNIX connection?
3. ovn-openflow-probe-interval is for the probe on TCP or both TCP
   and UNIX, if yes to #2.
Please comment here or in that thread.

> If you take ovn-20.06.01 then ovn-controller should not break often as
> there are many optimizations in the latest version to handle the DB
> changes more efficiently.

Will definitely give it a try and look forward to improvements.

> 
> Thanks
> Numan
> 
> >
> > >
> > > Thanks
> > > Numan
> > >
> > >
> > >
> > >
> > >
> > >       Thanks!
> > >
> > >       Tony
> > >
> > >       > -----Original Message-----
> > >       > From: discuss <ovs-discuss-bounces at openvswitch.org
> > > <mailto:ovs- discuss-bounces at openvswitch.org> > On Behalf Of Tony
> > >       > Liu
> > >       > Sent: Wednesday, August 5, 2020 2:48 PM
> > >       > To: ovs-discuss at openvswitch.org <mailto:ovs-
> > > discuss at openvswitch.org> ; ovs-dev at openvswitch.org <mailto:ovs-
> > > dev at openvswitch.org>
> > >       > Subject: [ovs-discuss] packet drop
> > >       >
> > >       > Hi,
> > >       >
> > >       > I am running ping from external to VM via OVN gateway.
> > >       > On the compute node, ICMP request packet is consistently
> > > coming into
> > >       > interface "ovn-gatewa-1". But there is about 10 out of 25
> > > packet loss on
> > >       > tap interface. It's like the switch pauses 10s after every
> 15s.
> > >       >
> > >       > Has anyone experiences such issue?
> > >       > Any advice how to look into it?
> > >       >
> > >       > ========
> > >       > 21fed09f-909e-4efc-b117-f5d5fcb636c9
> > >       >     Bridge br-int
> > >       >         fail_mode: secure
> > >       >         datapath_type: system
> > >       >         Port "ovn-gatewa-0"
> > >       >             Interface "ovn-gatewa-0"
> > >       >                 type: geneve
> > >       >                 options: {csum="true", key=flow,
> > > remote_ip="10.6.30.91"}
> > >       >                 bfd_status: {diagnostic="No Diagnostic",
> > > flap_count="1",
> > >       > forwarding="true", remote_diagnostic="No Diagnostic",
> > > remote_state=up,
> > >       > state=up}
> > >       >         Port "tap2588bb4e-35"
> > >       >             Interface "tap2588bb4e-35"
> > >       >         Port "ovn-gatewa-1"
> > >       >             Interface "ovn-gatewa-1"
> > >       >                 type: geneve
> > >       >                 options: {csum="true", key=flow,
> > > remote_ip="10.6.30.92"}
> > >       >                 bfd_status: {diagnostic="No Diagnostic",
> > > flap_count="1",
> > >       > forwarding="true", remote_diagnostic="No Diagnostic",
> > > remote_state=up,
> > >       > state=up}
> > >       >         Port "tap37f6b2d7-cc"
> > >       >             Interface "tap37f6b2d7-cc"
> > >       >         Port "tap2c4b3b0f-8b"
> > >       >             Interface "tap2c4b3b0f-8b"
> > >       >         Port "tap23245491-a4"
> > >       >             Interface "tap23245491-a4"
> > >       >         Port "tap51660269-2c"
> > >       >             Interface "tap51660269-2c"
> > >       >         Port "tap276cd1ef-e1"
> > >       >             Interface "tap276cd1ef-e1"
> > >       >         Port "tap138526d3-b3"
> > >       >             Interface "tap138526d3-b3"
> > >       >         Port "tapd1ae48a1-2d"
> > >       >             Interface "tapd1ae48a1-2d"
> > >       >         Port br-int
> > >       >             Interface br-int
> > >       >                 type: internal
> > >       >         Port "tapdd08f476-94"
> > >       >             Interface "tapdd08f476-94"
> > >       > ========
> > >       >
> > >       >
> > >       > Thanks!
> > >       >
> > >       > Tony
> > >       >
> > >       > _______________________________________________
> > >       > discuss mailing list
> > >       > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > >       > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > >       _______________________________________________
> > >       dev mailing list
> > >       dev at openvswitch.org <mailto:dev at openvswitch.org>
> > >       https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > >
> > >
> >
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >


More information about the discuss mailing list