[ovs-dev] [RFC PATCH] tunneling: Improving vxlan performance using DPDK flow director feature.

Jesse Gross jesse at kernel.org
Wed Mar 30 00:43:55 UTC 2016


On Tue, Mar 29, 2016 at 12:43 AM, Chandran, Sugesh
<sugesh.chandran at intel.com> wrote:
>> -----Original Message-----
>> From: Jesse Gross [mailto:jesse at kernel.org]
>> Sent: Friday, March 25, 2016 12:38 AM
>> To: Chandran, Sugesh <sugesh.chandran at intel.com>
>> Cc: dev at openvswitch.org
>> Subject: Re: [ovs-dev] [RFC PATCH] tunneling: Improving vxlan performance
>> using DPDK flow director feature.
>>  * Chaining together the multiple lookups used by tunnels on the assumption
>> that the outer VXLAN source port distinguishes the inner flow. This would
>> allow avoiding netdev_flow_key_equal_mf() a second time. This is definitely
>> not legal because the VXLAN source port is only capturing a small subset of
>> the total data that OVS is using.
>
> [Sugesh] From our analysis we found that optimizing one lookup give no
> significant performance boost when compared with the overhead. This is  due to the
> fact that the second netdev_flow_key_equal_mf() still need the tunnel information
> to match on a flow.  We found in our tests that most CPU cycles spends on extracting
> header fields from the packets than lookup.
>
> The proposal is to avoid the header field extraction by using an additional unique software
> flow ID to match on. The two flows for tunnel are marked with this ID when installing on the
> EMC. The hardware report this ID along with hash(to mitigate the hash collision in EMC)
> for every incoming packets that match on a hardware rule. This used in EMC
> along with hash to find the flow. Currently OVS compares  hash +key(from header fields)
> to match a flow. The inner flow uses the same unique ID and hardware flow flag to match
> on than the source port. We have modified the code little bit more, so that it saves the hardware
> id in the matching flow, for every emc_insert.

I think that the performance improvements look cool but unfortunately,
I just don't see how this can work.

There really isn't a way to avoid extracting the header fields in
software - I don't think that any NIC short of an NPU or other
programmable hardware has the capability to match on all of the fields
that OVS supports. Certainly, the UDP source port used by VXLAN and
other tunnel protocols does not contain all of the information and,
worse, it's controlled by a remote system. We can't trust the
information contained in it without further verification because OVS
flow rules are often used for security checks. I realize that in many
cases this will appear to work because for a flow represented by a
5-tuple many of the other fields will be the same. However, we can't
just make this assumption.

One possible exception to this rule is if we did an analysis on the
flows that are actually being used by OVS and only tried to extract
those fields. This is a pure software optimization that might have
similar effects to what you are observing here. This most likely makes
the most sense in the context of a BPF based datapath where the flow
extractor can be dynamically generated and compiled.

>> I'm not sure that I really see any advantage in using a Flow Director perfect
>> filter to return a software defined hash value compared to just using the RSS
>> hash directly as we are doing today. I think the main case where it would be
>> useful is if hardware wildcarding was used to skip the EMC altogether and its
>> size constraints. If that was done then I think that this would no longer be
>> specialized to VXLAN at all.
> [Sugesh] This may give performance improvement when we have
> large set of rules that overflows EMC. But for a typical use case where 80-90% rules hits EMC
> doesn’t get any performance benefit out of it. Please correct me if I am wrong here.
> The intention here is to optimize the tunneling performance in all the use cases.

To be honest, I think that last 10-20% may be more interesting. Up to
this point in time, the DPDK implementation in OVS has placed a lot of
emphasis on PPS throughput with a relatively small number of streams.
However, while this looks great on benchmarks, it doesn't necessarily
match real world use cases. Even worse, it tends to fall apart at the
worst possible times - like a DoS attack. If the NIC were able to
effectively enlarge the EMC to handle these cases then I think that
would be a huge boost to the usability of OVS on DPDK.



More information about the dev mailing list