[ovs-dev] [RFC PATCH] tunneling: Improving vxlan performance using DPDK flow director feature.

[Chandran, Sugesh]

Regards
_Sugesh


> -----Original Message-----
> From: Jesse Gross [mailto:jesse at kernel.org]
> Sent: Wednesday, March 30, 2016 1:44 AM
> To: Chandran, Sugesh <sugesh.chandran at intel.com>
> Cc: dev at openvswitch.org
> Subject: Re: [ovs-dev] [RFC PATCH] tunneling: Improving vxlan performance
> using DPDK flow director feature.
> 
> On Tue, Mar 29, 2016 at 12:43 AM, Chandran, Sugesh
> <sugesh.chandran at intel.com> wrote:
> >> -----Original Message-----
> >> From: Jesse Gross [mailto:jesse at kernel.org]
> >> Sent: Friday, March 25, 2016 12:38 AM
> >> To: Chandran, Sugesh <sugesh.chandran at intel.com>
> >> Cc: dev at openvswitch.org
> >> Subject: Re: [ovs-dev] [RFC PATCH] tunneling: Improving vxlan
> >> performance using DPDK flow director feature.
> >>  * Chaining together the multiple lookups used by tunnels on the
> >> assumption that the outer VXLAN source port distinguishes the inner
> >> flow. This would allow avoiding netdev_flow_key_equal_mf() a second
> >> time. This is definitely not legal because the VXLAN source port is
> >> only capturing a small subset of the total data that OVS is using.
> >
> > [Sugesh] From our analysis we found that optimizing one lookup give no
> > significant performance boost when compared with the overhead. This is
> > due to the fact that the second netdev_flow_key_equal_mf() still need
> > the tunnel information to match on a flow.  We found in our tests that
> > most CPU cycles spends on extracting header fields from the packets than
> lookup.
> >
> > The proposal is to avoid the header field extraction by using an
> > additional unique software flow ID to match on. The two flows for
> > tunnel are marked with this ID when installing on the EMC. The
> > hardware report this ID along with hash(to mitigate the hash collision
> > in EMC) for every incoming packets that match on a hardware rule. This
> > used in EMC along with hash to find the flow. Currently OVS compares
> > hash +key(from header fields) to match a flow. The inner flow uses the
> > same unique ID and hardware flow flag to match on than the source port.
> We have modified the code little bit more, so that it saves the hardware id in
> the matching flow, for every emc_insert.
> 
> I think that the performance improvements look cool but unfortunately, I
> just don't see how this can work.
> 
> There really isn't a way to avoid extracting the header fields in software - I
> don't think that any NIC short of an NPU or other programmable hardware
> has the capability to match on all of the fields that OVS supports. Certainly,
> the UDP source port used by VXLAN and other tunnel protocols does not
> contain all of the information and, worse, it's controlled by a remote system.
> We can't trust the information contained in it without further verification
> because OVS flow rules are often used for security checks. I realize that in
> many cases this will appear to work because for a flow represented by a 5-
> tuple many of the other fields will be the same. However, we can't just make
> this assumption.

[Sugesh] Totally agree with you. How about let the OVS program the NIC only if all the 
flow fields that can be supported in NIC. If the flow has more fields than the NIC can support, 
the rules will not get programmed on the hardware. 
I feel, If we make sure the rules in NIC can validate all the fields of that corresponding software flow, 
Its possible to avoid software header extraction on those packets. Any comments??

> 
> One possible exception to this rule is if we did an analysis on the flows that
> are actually being used by OVS and only tried to extract those fields. This is a
> pure software optimization that might have similar effects to what you are
> observing here. This most likely makes the most sense in the context of a
> BPF based datapath where the flow extractor can be dynamically generated
> and compiled.
This is really interesting, Will look at this option as well to see how much it can improve 
On tunneling.

> 
> >> I'm not sure that I really see any advantage in using a Flow Director
> >> perfect filter to return a software defined hash value compared to
> >> just using the RSS hash directly as we are doing today. I think the
> >> main case where it would be useful is if hardware wildcarding was
> >> used to skip the EMC altogether and its size constraints. If that was
> >> done then I think that this would no longer be specialized to VXLAN at all.
> > [Sugesh] This may give performance improvement when we have large set
> > of rules that overflows EMC. But for a typical use case where 80-90%
> > rules hits EMC doesn’t get any performance benefit out of it. Please correct
> me if I am wrong here.
> > The intention here is to optimize the tunneling performance in all the use
> cases.
> 
> To be honest, I think that last 10-20% may be more interesting. Up to this
> point in time, the DPDK implementation in OVS has placed a lot of emphasis
> on PPS throughput with a relatively small number of streams.
> However, while this looks great on benchmarks, it doesn't necessarily match
> real world use cases. Even worse, it tends to fall apart at the worst possible
> times - like a DoS attack. If the NIC were able to effectively enlarge the EMC
> to handle these cases then I think that would be a huge boost to the usability
> of OVS on DPDK.