[ovs-dev] [PATCH v2] tunneling: Improving tunneling performance using DPDK Rx checksum offloading feature.

Tue May 10 10:26:32 UTC 2016

Regards
_Sugesh

> -----Original Message-----
> From: Jesse Gross [mailto:jesse at kernel.org]
> Sent: Friday, May 6, 2016 5:00 PM
> To: Chandran, Sugesh <sugesh.chandran at intel.com>
> Cc: pravin shelar <pshelar at ovn.org>; ovs dev <dev at openvswitch.org>
> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling
> performance using DPDK Rx checksum offloading feature.
> 
> On Fri, May 6, 2016 at 1:13 AM, Chandran, Sugesh
> <sugesh.chandran at intel.com> wrote:
> >> -----Original Message-----
> >> From: Jesse Gross [mailto:jesse at kernel.org]
> >> Sent: Friday, May 6, 2016 1:58 AM
> >> To: Chandran, Sugesh <sugesh.chandran at intel.com>
> >> Cc: pravin shelar <pshelar at ovn.org>; ovs dev <dev at openvswitch.org>
> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling
> >> performance using DPDK Rx checksum offloading feature.
> >>
> >> On Thu, May 5, 2016 at 1:26 AM, Chandran, Sugesh
> >> <sugesh.chandran at intel.com> wrote:
> >> >> -----Original Message-----
> >> >> From: Jesse Gross [mailto:jesse at kernel.org]
> >> >> Sent: Wednesday, May 4, 2016 10:06 PM
> >> >> To: Chandran, Sugesh <sugesh.chandran at intel.com>
> >> >> Cc: pravin shelar <pshelar at ovn.org>; ovs dev
> <dev at openvswitch.org>
> >> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling
> >> >> performance using DPDK Rx checksum offloading feature.
> >> >>
> >> >> On Wed, May 4, 2016 at 8:58 AM, Chandran, Sugesh
> >> >> <sugesh.chandran at intel.com> wrote:
> >> >> >> -----Original Message-----
> >> >> >> From: Jesse Gross [mailto:jesse at kernel.org]
> >> >> >> Sent: Thursday, April 28, 2016 4:41 PM
> >> >> >> To: Chandran, Sugesh <sugesh.chandran at intel.com>
> >> >> >> Cc: pravin shelar <pshelar at ovn.org>; ovs dev
> >> <dev at openvswitch.org>
> >> >> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving
> >> >> >> tunneling performance using DPDK Rx checksum offloading feature.
> >> >> >
> >> >> >> That sounds great, thanks for following up. In the meantime, do
> >> >> >> you have any plans for transmit side checksum offloading?
> >> >> > [Sugesh] The vectorization on Tx side is getting disabled when
> >> >> > DPDK Tx
> >> >> checksum offload is enabled. This causes performance drop in OVS.
> >> >> > However We don’t find any such impact when enabling Rx checksum
> >> >> offloading(though this disables Rx vectorization).
> >> >>
> >> >> OK, I see. Does the drop in throughput cause performance to go
> >> >> below the baseline even for UDP tunnels with checksum traffic? (I
> >> >> guess small and large packets might have different results here.)
> >> >> Or is it that it reduce performance for unrelated traffic? If it's
> >> >> the latter case, can we find a way to use offloading conditionally?
> >> > [Sugesh] We tested for 64 byte UDP packet stream and found that the
> >> > performance is better when the offloading is turned off. This is
> >> > for any
> >> traffic through the port.
> >> > DPDK doesn’t support conditional offloading for now.
> >> > In other words DPDK can't do selective vector packet processing on a
> port.
> >> > As far as I know there are some technical difficulties to enable
> >> > offload + vectorization together in DPDK.
> >>
> >> My guess is the results might be different for larger packets since
> >> those cases will stress checksumming more and rx/tx routines less.
> >>
> >> In any case, I think this is an area that is worthwhile to continue
> investigating.
> >> My expectation is that tunneled packets with outer UDP checksums will
> >> be a use case that is hit increasingly frequently with OVS DPDK - for
> >> example, OVN will likely start exercising this soon.
> > [Sugesh]Totally agreed, I will do PHY-PHY, PHY-TUNNEL-PHY tests with
> > different size traffic
> > streams(64 Byte, 512, 1024, 1500) when checksum enabled/disabled and
> see the impact.
> > Is there any other traffic pattern/tests that we have to consider?
> 
> I think that should cover it pretty well. Thanks a lot!
[Sugesh] Please find below for the test results in different scenarios.

Native(Rx, Tx checksum offloading OFF)
Test			64 Bytes	128 Bytes	256 Bytes	512 Bytes	1500 bytes	Mix
PHY-PHY-BIDIR		9.2		8.445		4.528		2.349		0.822		6.205
PHY-VM-PHY-BIDIR	2.564		2.503		2.205		1.901		0.822		2.29
PHY-VXLAN-PHY	4.165		4.084		3.834		2.147		0.849		3.964

Rx Checksum ON/Rx Vector OFF						

Test			64 Bytes	128 Bytes	256 Bytes	512 Bytes	1500 bytes	Mix
PHY-PHY-BIDIR		9.12		8.445		4.528		2.349		0.822		6.205
PHY-VM-PHY-BIDIR	2.535		2.513		2.21		1.913		0.822		2.25
PHY-VXLAN-PHY	4.475		4.47		3.834		2.147		0.849		4.4

Tx Checksum ON/Tx Vector OFF						

Test			64 Bytes	128 Bytes	256 Bytes	512 Bytes	1500 bytes	Mix
PHY-PHY-BIDIR		7.89		7.935		4.528		2.349		0.822		6.205
PHY-VM-PHY-BIDIR	2.455		2.401		2.118		1.855		0.822		2.2
PHY-VXLAN-PHY	3.347		3.301		3.185		2.147		0.849		3.218

Rx, Tx Checksum ON/Rx, Tx Vector OFF						
Test			64 Bytes	128 Bytes	256 Bytes	512 Bytes	1500 bytes	Mix
PHY-PHY-BIDIR		7.75		7.74		4.528		2.349		0.822		6.205
PHY-VM-PHY-BIDIR	2.44		2.401		2.141		1.862		0.822		2.21
PHY-VXLAN-PHY	3.53		3.53		3.53		2.147		0.849		3.52

Findings:- 
1) For bigger packet sizes, (<= 256 bytes) the performance numbers are almost the same in all the scenarios.
2) Tx checksum offload doesn’t give any performance improvement in all the case. In fact in some scenarios, it shows low performance compared to native.
3) Rx checksum offload provide tunneling performance improvement mainly on small sized packets.

Please let me know your comments/thoughts on this?