[ovs-discuss] OVN /OVS openvswitch: ovs-system: deferred action limit reached, drop recirc action

Han Zhou hzhou at ovn.org
Wed Aug 4 18:02:45 UTC 2021


On Wed, Aug 4, 2021 at 6:41 AM Numan Siddique <numans at ovn.org> wrote:
>
> On Wed, Aug 4, 2021 at 4:17 AM Krzysztof Klimonda
> <kklimonda at syntaxhighlighted.com> wrote:
> >
> > Hi Ammad,
> >
> > (Re-adding ovs-discuss at openvswitch.org to CC to keep track of the
discussion)
> >
> > Thanks for testing it with SNAT enabled/disabled and verifying that it
seems to be related.
> >
> > As for the impact of this bug I have to say I'm unsure. I have
theorized that this could the cause for (or at least connected to) BFD
sessions being dropped between gateway chassises, but I couldn't really
validate it.
> >
> > My linked patch is pretty old and no longer applies cleanly on master,
but I'd be interested in getting some feedback from developers on whether
I'm even fixing the right thing.
>
> Hi Krzysztof,
>
> Your patch is in the "change requested" stage.  I see from the comment
> that the ddlog part of the code is missing.
>
> Seems like a valid case to me.  The issue is seen when the packet is
> destined to the router port IP right ?
>
> In the case of ovn-kubernetes, the router port IP is also used as a
> load balancer backend IP.
>
> Will your patch have any impact if the logical router has this load
> balancer configured ? (for the system test case you've added )
>
> ovn-nbctl lb-add lb1 172.16.1.254:90 192.168.1.100:90
> ovn-nbctl lr-lb-add R1 lb1
>
> Can you please repost the patch for further review.  It would be great
> if you can add ddlog code.  Or you can repost the patch
> and the ddlog part can be added if the reviewers are fine with the patch.
>
> Thanks
> Numan
>

Thanks Krzysztof, this is interesting. Could you share more on the root
cause since you debugged it - how did the loop happen? When a packet
destined to the SNAT IP hits the router ingress pipeline, what's the next
hop? How the L2 dst is populated for the dst IP and how is the packet
forwarded back to the router pipeline? How /32 IP (instead of a subnet) on
the SNAT config made a difference?

> >
> > Regards,
> > Krzysztof
> >
> > On Wed, Aug 4, 2021, at 09:02, Ammad Syed wrote:
> > > I am able to reproduce this issue with snat enabled network and
> > > accessing the snat IP from external network can reproduce this issue .
> > > If I keep snat disable, then I didn't see these logs in syslog.
> > >
> > > Ammad
> > >
> > > On Tue, Aug 3, 2021 at 6:39 PM Ammad Syed <syedammad83 at gmail.com>
wrote:
> > > > Thanks. Let me try to reproduce it with this way.
> > > >
> > > > Can you please advise if this will cause any trouble if we have
this bug in production? Any workaround to avoid this issue?
> > > >
> > > > Ammad
> > > >
> > > > On Tue, Aug 3, 2021 at 5:56 PM Krzysztof Klimonda <
kklimonda at syntaxhighlighted.com> wrote:
> > > >> Hi,
> > > >>
> > > >> To reproduce it (on openstack. although the issue does not seem to
be openstack-specific) I've created a network with SNAT enabled (which is
default) and set its external gateway to my external network. Next, I've
tried establishing TCP session from the outside to IP address assigned to
the router and checked dmesg on the chassis that the port is assigned to
for "ovs-system: deferred action limit reached, drop recirc action"
messages.
> > > >>
> > > >> Best Regards,
> > > >> Krzysztof
> > > >>
> > > >> On Tue, Aug 3, 2021, at 09:05, Ammad Syed wrote:
> > > >> > Hi Krzysztof,
> > > >> >
> > > >> > Yes I might be stuck in this issue. How can I check if there is
any
> > > >> > loop in lflow-list ?
> > > >> >
> > > >> > Ammad
> > > >> >
> > > >> > On Tue, Aug 3, 2021 at 2:14 AM Krzysztof Klimonda
> > > >> > <kklimonda at syntaxhighlighted.com> wrote:
> > > >> > > Hi,
> > > >> > >
> > > >> > > Not sure if it's related, but I've seen this bug in ovn 20.12
release, where routing loop was related to flows created to handle SNAT,
I've sent an RFC patch few months back but didn't really have time to
follow up on it since then to get some feedback:
https://www.mail-archive.com/ovs-dev@openvswitch.org/msg53195.html
> > > >> > > I was planning on re-testing it with 21.06 release and follow
up on the patch.
> > > >> > >
> > > >> > > On Mon, Aug 2, 2021, at 21:31, Han Zhou wrote:
> > > >> > > >
> > > >> > > >
> > > >> > > > On Mon, Aug 2, 2021 at 5:07 AM Ammad Syed <
syedammad83 at gmail.com> wrote:
> > > >> > > > >
> > > >> > > > > Hello,
> > > >> > > > >
> > > >> > > > > I am using openstack with OVN 20.12 and OVS 2.15.0 on
ubuntu 20.04. I am using geneve tenant network and vlan provider network.
> > > >> > > > >
> > > >> > > > > I am continuously getting below messages in my dmesg logs
continuously on compute node 1 only the other two compute nodes have no
such messages.
> > > >> > > > >
> > > >> > > > > [275612.826698] openvswitch: ovs-system: deferred action
limit reached, drop recirc action
> > > >> > > > > [275683.750343] openvswitch: ovs-system: deferred action
limit reached, drop recirc action
> > > >> > > > > [276102.200772] openvswitch: ovs-system: deferred action
limit reached, drop recirc action
> > > >> > > > > [276161.575494] openvswitch: ovs-system: deferred action
limit reached, drop recirc action
> > > >> > > > > [276210.262524] openvswitch: ovs-system: deferred action
limit reached, drop recirc action
> > > >> > > > >
> > > >> > > > > I have tried by reinstalling (OS everything) compute node
1 but still having same errors.
> > > >> > > > >
> > > >> > > > > Need your advise.
> > > >> > > > >
> > > >> > > > > --
> > > >> > > > > Regards,
> > > >> > > > >
> > > >> > > > >
> > > >> > > > > Syed Ammad Ali
> > > >> > > > > _______________________________________________
> > > >> > > > > discuss mailing list
> > > >> > > > > discuss at openvswitch.org
> > > >> > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > > >> > > >
> > > >> > > > Hi Syed,
> > > >> > > >
> > > >> > > > Could you check if you have routing loops (i.e. a packet
being routed
> > > >> > > > back and forth between logical routers infinitely) in your
logical
> > > >> > > > topology?
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Han
> > > >> > > > _______________________________________________
> > > >> > > > discuss mailing list
> > > >> > > > discuss at openvswitch.org
> > > >> > > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > > >> > > >
> > > >> > >
> > > >> > >
> > > >> > > --
> > > >> > >   Krzysztof Klimonda
> > > >> > >   kklimonda at syntaxhighlighted.com
> > > >> > > _______________________________________________
> > > >> > > discuss mailing list
> > > >> > > discuss at openvswitch.org
> > > >> > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> > > >> >
> > > >> >
> > > >> > --
> > > >> > Regards,
> > > >> >
> > > >> >
> > > >> > Syed Ammad Ali
> > > >>
> > > >>
> > > >> --
> > > >>   Krzysztof Klimonda
> > > >>   kklimonda at syntaxhighlighted.com
> > > > --
> > > > Regards,
> > > >
> > > >
> > > > Syed Ammad Ali
> > >
> > >
> > > --
> > > Regards,
> > >
> > >
> > > Syed Ammad Ali
> >
> >
> > --
> >   Krzysztof Klimonda
> >   kklimonda at syntaxhighlighted.com
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20210804/f32fa28a/attachment.html>


More information about the discuss mailing list