[ovs-dev] request for review: protection against kernel flow trashing

Jesse Gross jesse at nicira.com
Wed Apr 3 14:48:19 UTC 2013


On Wed, Apr 3, 2013 at 6:29 AM, George Shuklin <george.shuklin at gmail.com> wrote:
> Good day.
>
> Few weeks ago I have reported a serious flaw in OVS: specially crafted flood
> or even a legal traffic with a large number of tcp connections from
> different sources can cause a denial of service.
>
> Initially I thought it's somehow caused by 'slow' normal mode rules, but
> further research have shown that even a single rule 'priority=10,
> actions=drop' can cause absolutely unacceptable CPU load by ovs-vswitchd.
> Flood (or just large number of legal connections) with rate about 5k new
> flows per second cause 100% CPU load, huge latency spikes and overall system
> performance degradation.
>
> This issue is especially harmful for XenServer and XCP, which use OVS for
> management/SAN traffic. Under flood-load about 30-40Mb/s, it cause horrible
> network delays on all bridges, virtual machines starts to get IO timeouts
> and propagate them as IO errors to VM userspace. Even neglecting SAN
> traffic, 30Mb/s from Internet can cause outage of server with 1G, 10G
> interfaces. This is ridiculous.
>
> I've done some deep research and found the reason:
>
> Every packet is inspected in kernel space by "extract_flow" function.
> Extracted key is hashed and used to search for corresponding rule in the
> flows table. If rule is found, it's used. If not - packet sent to userspace
> ovs-switchd daemon for inspection. This process is very slow compare to
> kernel-only mode. Normally OVS kernel module can process up to 100kpps per
> single VIF (and I think this is a limitation of xen's netback/front driver,
> not OVS), but passing those packets to userspace cause performance drops to
> 3-5-7kpps. And not 'per single VIF', but 'per host'.
>
> The problem is that packets with different source IP, source/destination
> TCP/UDP ports, different TTL, etc... produce different hashes. Any change of
> any field of any header on any level results in new key and additional
> "slow" queries to ovs-vswitchd. Large number of TCP/UDP sessions => many
> different keys => slow processing rate.
>
> That over-deep packet inspection is the reason why performance of OVS is
> very inconsistent on different load types. Under one kind of load it shows
> excellent performance, many times faster than brtools, but dropping to over
> 10 times slower on the other load type.
>
> To fix that issue I've decided to 'cut off' some of the OVS key extraction
> functionality. I understand that this will make OVS a "cripple"(not a fully
> compatible open flow switch), but few weeks of testing in labs and two weeks
> in product showed that patch [1] significantly reduces DoS vulnerability and
> fixes some strange latency fluctuation under heavy load.
>
> Patch is built on few assumptions:
>
> 1) OVS runs under XenServer/XCP, therefore port 0 is always connected to
> xenbrX, port 1 connected to physical interface (eth) and all ports above 1
> is VMs ports.
> 2) Amount of source mac/ip from guest VMs is limited
> 3) Support for antispoofing rules (restriction for VMs to use only allowed
> source MAC/IP addresses)
> 4) No specific interest in xenbr/eth traffic.
>
> Changes:
>
> 1) ARP and ipv6 traffic processing is left unchanged (no ipv6 DoS protection
> for now, I will bother about that only after first serious activity)
> 2) For xenbr/eth interfaces: no source/destination mac/IP addresses are
> added to flow.
> 3) For interfaces >1 (VMs): only source IP and source MAC is added to key.
> 4) No TTL, QoS, etc fields are taken into account.
>
> If it's possible I'd like to hear comments on these changes. Thanks.

I'm not sure what you're hoping to hear.  The solution is obviously
specific to your situation and not something that we can apply.



More information about the dev mailing list