[ovs-discuss] [OVN] Too many resubmits for packets coming from "external" network

Tue Sep 29 10:14:18 UTC 2020

On Tue, Sep 29, 2020 at 11:14 AM Krzysztof Klimonda <
kklimonda at syntaxhighlighted.com> wrote:

> On Tue, Sep 29, 2020, at 10:40, Dumitru Ceara wrote:
> > On 9/29/20 12:42 AM, Krzysztof Klimonda wrote:
> > > Hi Dumitru,
> > >
> > > This cluster is IPv4-only for now - there are no IPv6 networks defined
> at all - overlay or underlay.
> > >
> > > However, once I increase a number of routers to ~250, a similar
> behavior can be observed when I send ARP packets for non-existing IPv4
> addresses. The following warnings will flood ovs-vswitchd.log for every
> address not known to OVN when I run `fping -g 192.168.0.0/16`
> <http://192.168.0.0/16>:
> > >
> > > ---8<---8<---8<---
> > > 2020-09-28T22:26:40.967Z|21996|ofproto_dpif_xlate(handler6)|WARN|over
> 4096 resubmit actions on bridge br-int while processing
> arp,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:75:38:be,dl_dst=ff:ff:ff:ff:ff:ff,arp_spa=192.168.0.1,arp_tpa=192.168.0.35,arp_op=1,arp_sha=fa:16:3e:75:38:be,arp_tha=00:00:00:00:00:00
> > > ---8<---8<---8<---
> > >
> > > This is even a larger concern for me, as some of our clusters would be
> exposed to the internet where we can't easily prevent scanning of an entire
> IP range.
> > >
> > > Perhaps this is something that should be handled differently for
> traffic coming from external network? Is there any reason why OVN is not
> dropping ARP requests and IPv6 ND for IP addresses it knows nothing about?
> Or maybe OVN should drop most of BUM traffic on external network in
> general? I think all this network is used for is SNAT and/or SNAT+DNAT for
> overlay networks.
> > >
> >
> > Ok, so I guess we need a combination of the existing broadcast domain
> > limiting options:
> >
> > 1. send ARP/NS packets only to router ports that own the target IP
> address.
> > 2. flood IPv6 ND RS packets only to router ports with IPv6 addresses
> > configured and ipv6_ra_configs.address_mode set.
> > 3. according to the logical switch multicast configuration either flood
> > unkown IP multicast or forward it only to hosts that registered for the
> > IP multicast group.
> > 4. drop all other BUM traffic.
> >
> > From the above, 1 and 3 are already implemented. 2 is what I suggested
> > earlier. 4 would probably turn out to be configuration option that needs
> > to be explicitly enabled on the logical switch connected to the external
> > network.
> >
> > Would this work for you?
>
> I believe it would work for me, although it may be a good idea to consult
> with neutron developers and see if they have any input on that.
>

I think that's a good plan. Implementing 4) via a configuration option
sounds smart. From an OpenStack point of view, I think that as all the
ports are known, we can just have it on by default.
We need to make sure it works for 'edge' cases like virtual ports, load
balancers and subports (ports with a parent port and a tag) but the idea
sounds great to me.

Thanks folks for the discussion!

>
> >
> > Thanks,
> > Dumitru
> >
> > > -- Krzysztof Klimonda kklimonda at syntaxhighlighted.com On Mon, Sep 28,
> > > 2020, at 21:14, Dumitru Ceara wrote:
> > >> On 9/28/20 5:33 PM, Krzysztof Klimonda wrote:
> > >>> Hi,
> > >>>
> > >> Hi Krzysztof,
> > >>
> > >>> We're still doing some scale tests of OpenStack ussuri with ml2/ovn
> driver. We've deployed 140 virtualized compute nodes, and started creating
> routers that share single external network between them. Additionally, each
> router is connected to a private network.
> > >>> Previously[1] we hit a problem of too many logical flows being
> generated per router connected to the same "external" network - this put
> too much stress on ovn-controller and ovs-vswitchd on compute nodes, and
> we've applied a patch[2] to limit a number of logical flows created per
> router.
> > >>> After we dealt with that we've done more testing and created 200
> routers connected to single external network. After that we've noticed the
> following logs in ovs-vswitchd.log:
> > >>>
> > >>> ---8<---8<---8<---
> > >>>
> 2020-09-28T11:10:18.938Z|18401|ofproto_dpif_xlate(handler9)|WARN|over 4096
> resubmit actions on bridge br-int while processing
> icmp6,in_port=1,vlan_tci=0x0000,dl_src=fa:16:3e:9b:77:c3,dl_dst=33:33:00:00:00:02,ipv6_src=fe80::f816:3eff:fe9b:77c3,ipv6_dst=ff02::2,ipv6_label=0x2564e,nw_tos=0,nw_ecn=0,nw_ttl=255,icmp_type=133,icmp_code=0
> > >>> ---8<---8<---8<---
> > >>>
> > >>> That starts happening after I create ~178 routers connected to the
> same external network.
> > >>>
> > >>> IPv6 RS ICMP packets are coming from the external network - that's
> due to the fact that all virtual compute nodes have IPv6 address on their
> interface used for the external network and are trying to discover a
> gateway. That's by accident, and we can remove IPv6 address from that
> interface, however I'm worried that it would just hide some bigger issue
> with flows generated by OVN.
> > >>>
> > >> Is this an IPv4 cluster; are there IPv6 addresses configured on the
> > >> logical router ports connected to the external network?
> > >>
> > >> If there are IPv6 addresses, do the logical router ports connected to
> > >> the external network have
> > >> Logical_Router_Port.ipv6_ra_configs.address_mode set?
> > >>
> > >> If not, we could try to enhance the broadcast domain limiting code in
> > >> OVN [3] to also limit sending router solicitations only to router
> ports
> > >> with address_mode configured.
> > >>
> > >> Regards,
> > >> Dumitru
> > >>
> > >> [3]
> > >>
> https://github.com/ovn-org/ovn/blob/20a20439219493f27eb222617f045ba54c95ebfc/northd/ovn-northd.c#L6424
> > >>
> > >>> software stack:
> > >>>
> > >>> ovn: 20.06.2
> > >>> ovs: 2.13.1
> > >>> neutron: 16.1.0
> > >>>
> > >>> [1]
> http://lists.openstack.org/pipermail/openstack-discuss/2020-September/017370.html
> > >>> [2] https://review.opendev.org/#/c/752678/
> > >>>
> > >>
> >
> >
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200929/e561cabd/attachment.html>