[ovs-dev] upcall fairness

[Dustin Lundquist]

I have some concerns about the fairness of upcalls and flow cache
evictions in the kernel dataplane. My motivation for investigating this
is based on my experience operating a hardware vendor's solution which
use a similar method: caching routing decisions as flows after the first
packet was punted to software. Operating a network using such hardware,
we found a single host could generate traffic which would force cache
evictions adversely effecting other hosts' traffic.

As OVS uses a similar approach, I wanted to test how well it handles a
malicious or compromised host connected to one of its ports. To test
this I used Devstack with networking-ovn and created three neutron probe
namespaces on a single network. Within one of the namespaces I used the
Linux pktgen[1] kernel module to generate a large number of MAC
flows[2]. I found ovs-vswitchd exhibited high CPU utilization, evidence
of datapath flow evictions in latencies of a single ping test:

> ubuntu at devstack-ovn:~$ sudo ip netns exec qprobe-1739b61e-4085-43b5-bd4e-05051a11d996 ping -c 10 10.0.0.2
> PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.                                                           
> 64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=11.1 ms                                                   
> 64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.091 ms                                                  
> 64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=2.28 ms                                                   
> 64 bytes from 10.0.0.2: icmp_seq=4 ttl=64 time=2.69 ms                                                   
> 64 bytes from 10.0.0.2: icmp_seq=5 ttl=64 time=0.086 ms                                                  
> 64 bytes from 10.0.0.2: icmp_seq=6 ttl=64 time=0.081 ms                                                  
> 64 bytes from 10.0.0.2: icmp_seq=7 ttl=64 time=0.086 ms                                                  
> 64 bytes from 10.0.0.2: icmp_seq=8 ttl=64 time=0.085 ms                                                  
> 64 bytes from 10.0.0.2: icmp_seq=9 ttl=64 time=0.085 ms                                                  
> 64 bytes from 10.0.0.2: icmp_seq=10 ttl=64 time=0.084 ms                                                 
>                                                                                                          
> --- 10.0.0.2 ping statistics ---                                                                         
> 10 packets transmitted, 10 received, 0% packet loss, time 9004ms                                         
> rtt min/avg/max/mdev = 0.081/1.671/11.134/3.295 ms                                                       

Are their any existing or planned methods to ensure traffic from tenant
ports is handled fairly, preventing traffic received on a single port
from evicting flows from all other ports? Obviously uplink ports would
need to be permitted a larger portion of flows.


-Dustin


[1] https://www.kernel.org/doc/Documentation/networking/pktgen.txt
[2]
https://github.com/dlundquist/openstack-scripts/blob/master/scripts/dp_test.sh