[ovs-discuss] Megaflow Inspection

Matan Rosenberg matan129 at gmail.com
Tue Jan 7 20:44:57 UTC 2020


Acutally, I do think I have a megaflow (or other caching) issue.

We use OVS for L2 packet forwarding; that is, given a packet, we don't need
OVS to look at other protocols beyond the Ethernet layer.
Additionally, we use VXLAN to establish L2 overlay networks across multiple
OVS servers.

Just to make thing clear, these are some typical flow rules that you might
see on a bridge:

- in_port=1,actions=2,3
- in_port=42,actions=FLOOD
- actions=NORMAL

No IP matching, conntrack, etc.

We're experiencing severe performance issues with OVS - in this use case,
it cannot handle more than couple thousand packets/s.
After some exploring, I've noticed that the installed megaflows try to
match on fields that are not present in the rules, apparently for no reason.
Here's a complete example to reproduce, using OVS 2.12.0:

# ip link add dev a-blue type veth peer name a-red
# ip link add dev b-blue type veth peer name b-red

# ovs-vsctl add-br br0
# ovs-vsctl add-port br0 a-blue
# ovs-vsctl add-port br0 b-blue

# ovs-ofctl del-flows br0
# ovs-ofctl add-flow br0 in_port=a-blue,actions=b-blue
# ovs-ofctl add-flow br0 in_port=b-blue,actions=a-blue

After injecting ~100 random packets (IP, IPv6, TCP, UDP, ARP with random
addresses) to one of the red interfaces (with https://pastebin.com/Y6dPFCKJ),
these are the installed flows:
# ovs-dpctl dump-flows
recirc_id(0),in_port(2),eth(),eth_type(0x0806), packets:54, bytes:2268,
used:1.337s, actions:3
recirc_id(0),in_port(2),eth(),eth_type(0x86dd),ipv6(frag=no), packets:28,
bytes:1684, used:1.430s, flags:S, actions:3
recirc_id(0),in_port(2),eth(),eth_type(0x0800),ipv4(frag=no), packets:15,
bytes:610, used:1.270s, flags:S, actions:3

As you can see, for some reason, OVS had split the single relevant OpenFlow
rule to three separate megaflows, one for each eth_type (and even other
fields - IP fragmentation?).
In my production scenario, the packets are even more diversified, and we
see OVS installing flows which match on even more fields, including
specific Ethernet and IP addresses.

This leads to a large number of flows that have extremely low hit rate -
each flow handles not more than ~100 packets (!) during its entire lifetime.

We suspect that this causes the performance peanalty; either
1) The EMC/megaflow table is full, so vswitchd upcalls are all over the
place, or
2) The huge number of inefficient megaflows leads to terrible lookup times
in the in-kernel megaflow table itslef (due to large number of masks, etc.)

In short: how can I just make OVS oblivious to these fields? Why does it
try to match on irrlevant fields?

Thanks,
Matan

On Thu, 19 Dec 2019 at 02:19, Ben Pfaff <blp at ovn.org> wrote:

> On Wed, Dec 18, 2019 at 04:26:09PM +0200, Matan Rosenberg wrote:
> > Hey,
> > I'm using ovs for L2 tunneling (across lots of endpoints and several ovs
> > nodes), and experiencing some unexpected performance issues (can't get
> more
> > that a couple thousand packets/s on rather beefy servers).
> >
> > I've come across the talk about megaflow issues in OVSFall2018, and I
> > suspect that it might be applicable to my situation.
> >
> >  Is there any way to view the megaflows and masks currently used by the
> > kernel datapath?
>
> I can't see how this would be a megaflow problem, but either way:
>
> ovs-dpctl dump-flows
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20200107/550bab88/attachment.html>


More information about the discuss mailing list