[ovs-discuss] [ovn] Broken ovs localport flow for ovnmeta namespaces created by neutron

Thu Feb 4 14:05:49 UTC 2021

Hello Numan,

But the latest package in CentOS NFV SIG repo is built/uploaded 16th Dec
2020.
Are you sure it contains the fix?

czw., 4 lut 2021 o 13:53 Numan Siddique <numans at ovn.org> napisał(a):

> On Thu, Feb 4, 2021 at 5:04 PM Michał Nasiadka <mnasiadka at gmail.com>
> wrote:
> >
> > Hello,
> >
> > I've been hitting the same issue with OVN 20.09 from CentOS NFV SIG repo
> - is there a chance to backport this change to 20.09?
>
> Its already backported a while ago -
>
> https://github.com/ovn-org/ovn/commit/ab1e46d7ec7a2ba80d68fbf6a45d09eef3269541
>
> Numan
>
> >
> > czw., 17 gru 2020 o 11:05 Daniel Alvarez Sanchez <dalvarez at redhat.com>
> napisał(a):
> >>
> >>
> >>
> >> On Tue, Dec 15, 2020 at 11:39 AM Krzysztof Klimonda <
> kklimonda at syntaxhighlighted.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Just as a quick update - I've updated our ovn version to 20.12.0
> snapshot (d8bc0377c) and so far the problem hasn't yet reoccurred after
> over 24 hours of tempest testing.
> >>
> >>
> >> We could reproduce the issue with 20.12 and master. Also this is not
> related exclusively to localports but to any port potentially.
> >> Dumitru posted a fix for this:
> >>
> >>
> http://patchwork.ozlabs.org/project/ovn/patch/1608197000-637-1-git-send-email-dceara@redhat.com/
> >>
> >> Thanks!
> >> daniel
> >>>
> >>>
> >>> Best Regards,
> >>> -Chris
> >>>
> >>>
> >>> On Tue, Dec 15, 2020, at 11:13, Daniel Alvarez Sanchez wrote:
> >>>
> >>> Hey Krzysztof,
> >>>
> >>> On Fri, Nov 20, 2020 at 1:17 PM Krzysztof Klimonda <
> kklimonda at syntaxhighlighted.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> Doing some tempest runs on our pre-prod environment (stable/ussuri
> with ovn 20.06.2 release) I've noticed that some network connectivity tests
> were failing randomly. I've reproduced that by conitnously rescuing and
> unrescuing instance - network connectivity from and to VM works in general
> (dhcp is fine, access from outside is fine), however VM has no access to
> its metadata server (via 169.254.169.254 ip address). Tracing packet from
> VM to metadata via:
> >>>
> >>> ----8<----8<----8<----
> >>> ovs-appctl ofproto/trace br-int
> in_port=tapa489d406-91,dl_src=fa:16:3e:2c:b0:fd,dl_dst=fa:16:3e:8b:b5:39
> >>> ----8<----8<----8<----
> >>>
> >>> ends with
> >>>
> >>> ----8<----8<----8<----
> >>> 65. reg15=0x1,metadata=0x97e, priority 100, cookie 0x15ec4875
> >>>     output:1187
> >>>      >> Nonexistent output port
> >>> ----8<----8<----8<----
> >>>
> >>> And I can verify that there is no flow for the actual ovnmeta tap
> interface (tap67731b0a-c0):
> >>>
> >>> ----8<----8<----8<----
> >>> # docker exec -it openvswitch_vswitchd ovs-ofctl dump-flows br-int
> |grep -E output:'("tap67731b0a-c0"|1187)'
> >>>  cookie=0x15ec4875, duration=1868.378s, table=65, n_packets=524,
> n_bytes=40856, priority=100,reg15=0x1,metadata=0x97e actions=output:1187
> >>> #
> >>> ----8<----8<----8<----
> >>>
> >>> From ovs-vswitchd.log it seems the interface tap67731b0a-c0 was added
> with index 1187, then deleted, and re-added with index 1189 - that's
> probably due to the fact that that is the only VM in that network and I'm
> constantly hard rebooting it via rescue/unrescue:
> >>>
> >>> ----8<----8<----8<----
> >>> 2020-11-20T11:41:18.347Z|08043|bridge|INFO|bridge br-int: added
> interface tap67731b0a-c0 on port 1187
> >>> 2020-11-20T11:41:30.813Z|08044|bridge|INFO|bridge br-int: deleted
> interface tapa489d406-91 on port 1186
> >>> 2020-11-20T11:41:30.816Z|08045|bridge|WARN|could not open network
> device tapa489d406-91 (No such device)
> >>> 2020-11-20T11:41:31.040Z|08046|bridge|INFO|bridge br-int: deleted
> interface tap67731b0a-c0 on port 1187
> >>> 2020-11-20T11:41:31.044Z|08047|bridge|WARN|could not open network
> device tapa489d406-91 (No such device)
> >>> 2020-11-20T11:41:31.050Z|08048|bridge|WARN|could not open network
> device tapa489d406-91 (No such device)
> >>> 2020-11-20T11:41:31.235Z|08049|connmgr|INFO|br-int<->unix#31: 2069
> flow_mods in the last 43 s (858 adds, 814 deletes, 397 modifications)
> >>> 2020-11-20T11:41:33.057Z|08050|bridge|INFO|bridge br-int: added
> interface tapa489d406-91 on port 1188
> >>> 2020-11-20T11:41:33.582Z|08051|bridge|INFO|bridge br-int: added
> interface tap67731b0a-c0 on port 1189
> >>> 2020-11-20T11:42:31.235Z|08052|connmgr|INFO|br-int<->unix#31: 168
> flow_mods in the 2 s starting 59 s ago (114 adds, 10 deletes, 44
> modifications)
> >>> ----8<----8<----8<----
> >>>
> >>> Once I restart ovn-controller it recalculates local ovs flows and the
> problem is fixed so I'm assuming it's a local problem and not related to NB
> and SB databases.
> >>>
> >>>
> >>> I have seen exactly the same which with 20.09, for the same port input
> and output ofports do not match:
> >>>
> >>> bash-4.4# ovs-ofctl dump-flows br-int table=0 | grep 745
> >>>  cookie=0x38937d8e, duration=40387.372s, table=0, n_packets=1863,
> n_bytes=111678, idle_age=1, priority=100,in_port=745
> actions=load:0x4b->NXM_NX_REG13[],load:0x6a->NXM_NX_REG11[],load:0x69->NXM_NX_REG12[],load:0x18d->OXM_OF_METADATA[],load:0x1->NXM_NX_REG14[],resubmit(,8)
> >>>
> >>>
> >>> bash-4.4# ovs-ofctl dump-flows br-int table=65 | grep 8937d8e
> >>>  cookie=0x38937d8e, duration=40593.699s, table=65, n_packets=1848,
> n_bytes=98960, idle_age=2599, priority=100,reg15=0x1,metadata=0x18d
> actions=output:737
> >>>
> >>>
> >>> In table=0, the ofport is fine (745) but in the output stage it is
> using a different one (737).
> >>>
> >>> By checking the OVS database transaction history, that port, at some
> point, had the id 737:
> >>>
> >>> record 6516: 2020-12-14 22:22:54.184
> >>>
> >>>   table Interface row "tap71a5dfc1-10" (073801e2):
> >>>     ofport=737
> >>>   table Open_vSwitch row 1d9566c8 (1d9566c8):
> >>>     cur_cfg=2023
> >>>
> >>> So it looks like ovn-controller is not updating the ofport in the
> physical flows for the output stage.
> >>>
> >>> We'll try to figure out if this happens also in master.
> >>>
> >>> Thanks,
> >>> daniel
> >>>
> >>>
> >>> --
> >>>   Krzysztof Klimonda
> >>>   kklimonda at syntaxhighlighted.com
> >>> _______________________________________________
> >>> discuss mailing list
> >>> discuss at openvswitch.org
> >>> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >>>
> >>>
> >> _______________________________________________
> >> discuss mailing list
> >> discuss at openvswitch.org
> >> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
> >
> >
> > --
> > Michał Nasiadka
> > mnasiadka at gmail.com
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>

-- 
Michał Nasiadka
mnasiadka at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20210204/2ddb62d5/attachment-0001.html>