[ovs-dev] [PATCH v2] tunneling: Improving tunneling performance using DPDK Rx checksum offloading feature.
Jesse Gross
jesse at kernel.org
Wed May 11 01:31:51 UTC 2016
On Tue, May 10, 2016 at 3:26 AM, Chandran, Sugesh
<sugesh.chandran at intel.com> wrote:
>> -----Original Message-----
>> From: Jesse Gross [mailto:jesse at kernel.org]
>> Sent: Friday, May 6, 2016 5:00 PM
>> To: Chandran, Sugesh <sugesh.chandran at intel.com>
>> Cc: pravin shelar <pshelar at ovn.org>; ovs dev <dev at openvswitch.org>
>> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling
>> performance using DPDK Rx checksum offloading feature.
>>
>> On Fri, May 6, 2016 at 1:13 AM, Chandran, Sugesh
>> <sugesh.chandran at intel.com> wrote:
>> >> -----Original Message-----
>> >> From: Jesse Gross [mailto:jesse at kernel.org]
>> >> Sent: Friday, May 6, 2016 1:58 AM
>> >> To: Chandran, Sugesh <sugesh.chandran at intel.com>
>> >> Cc: pravin shelar <pshelar at ovn.org>; ovs dev <dev at openvswitch.org>
>> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling
>> >> performance using DPDK Rx checksum offloading feature.
>> >>
>> >> On Thu, May 5, 2016 at 1:26 AM, Chandran, Sugesh
>> >> <sugesh.chandran at intel.com> wrote:
>> >> >> -----Original Message-----
>> >> >> From: Jesse Gross [mailto:jesse at kernel.org]
>> >> >> Sent: Wednesday, May 4, 2016 10:06 PM
>> >> >> To: Chandran, Sugesh <sugesh.chandran at intel.com>
>> >> >> Cc: pravin shelar <pshelar at ovn.org>; ovs dev
>> <dev at openvswitch.org>
>> >> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving tunneling
>> >> >> performance using DPDK Rx checksum offloading feature.
>> >> >>
>> >> >> On Wed, May 4, 2016 at 8:58 AM, Chandran, Sugesh
>> >> >> <sugesh.chandran at intel.com> wrote:
>> >> >> >> -----Original Message-----
>> >> >> >> From: Jesse Gross [mailto:jesse at kernel.org]
>> >> >> >> Sent: Thursday, April 28, 2016 4:41 PM
>> >> >> >> To: Chandran, Sugesh <sugesh.chandran at intel.com>
>> >> >> >> Cc: pravin shelar <pshelar at ovn.org>; ovs dev
>> >> <dev at openvswitch.org>
>> >> >> >> Subject: Re: [ovs-dev] [PATCH v2] tunneling: Improving
>> >> >> >> tunneling performance using DPDK Rx checksum offloading feature.
>> >> >> >
>> >> >> >> That sounds great, thanks for following up. In the meantime, do
>> >> >> >> you have any plans for transmit side checksum offloading?
>> >> >> > [Sugesh] The vectorization on Tx side is getting disabled when
>> >> >> > DPDK Tx
>> >> >> checksum offload is enabled. This causes performance drop in OVS.
>> >> >> > However We don’t find any such impact when enabling Rx checksum
>> >> >> offloading(though this disables Rx vectorization).
>> >> >>
>> >> >> OK, I see. Does the drop in throughput cause performance to go
>> >> >> below the baseline even for UDP tunnels with checksum traffic? (I
>> >> >> guess small and large packets might have different results here.)
>> >> >> Or is it that it reduce performance for unrelated traffic? If it's
>> >> >> the latter case, can we find a way to use offloading conditionally?
>> >> > [Sugesh] We tested for 64 byte UDP packet stream and found that the
>> >> > performance is better when the offloading is turned off. This is
>> >> > for any
>> >> traffic through the port.
>> >> > DPDK doesn’t support conditional offloading for now.
>> >> > In other words DPDK can't do selective vector packet processing on a
>> port.
>> >> > As far as I know there are some technical difficulties to enable
>> >> > offload + vectorization together in DPDK.
>> >>
>> >> My guess is the results might be different for larger packets since
>> >> those cases will stress checksumming more and rx/tx routines less.
>> >>
>> >> In any case, I think this is an area that is worthwhile to continue
>> investigating.
>> >> My expectation is that tunneled packets with outer UDP checksums will
>> >> be a use case that is hit increasingly frequently with OVS DPDK - for
>> >> example, OVN will likely start exercising this soon.
>> > [Sugesh]Totally agreed, I will do PHY-PHY, PHY-TUNNEL-PHY tests with
>> > different size traffic
>> > streams(64 Byte, 512, 1024, 1500) when checksum enabled/disabled and
>> see the impact.
>> > Is there any other traffic pattern/tests that we have to consider?
>>
>> I think that should cover it pretty well. Thanks a lot!
> [Sugesh] Please find below for the test results in different scenarios.
>
> Native(Rx, Tx checksum offloading OFF)
> Test 64 Bytes 128 Bytes 256 Bytes 512 Bytes 1500 bytes Mix
> PHY-PHY-BIDIR 9.2 8.445 4.528 2.349 0.822 6.205
> PHY-VM-PHY-BIDIR 2.564 2.503 2.205 1.901 0.822 2.29
> PHY-VXLAN-PHY 4.165 4.084 3.834 2.147 0.849 3.964
>
>
> Rx Checksum ON/Rx Vector OFF
>
> Test 64 Bytes 128 Bytes 256 Bytes 512 Bytes 1500 bytes Mix
> PHY-PHY-BIDIR 9.12 8.445 4.528 2.349 0.822 6.205
> PHY-VM-PHY-BIDIR 2.535 2.513 2.21 1.913 0.822 2.25
> PHY-VXLAN-PHY 4.475 4.47 3.834 2.147 0.849 4.4
>
>
>
> Tx Checksum ON/Tx Vector OFF
>
> Test 64 Bytes 128 Bytes 256 Bytes 512 Bytes 1500 bytes Mix
> PHY-PHY-BIDIR 7.89 7.935 4.528 2.349 0.822 6.205
> PHY-VM-PHY-BIDIR 2.455 2.401 2.118 1.855 0.822 2.2
> PHY-VXLAN-PHY 3.347 3.301 3.185 2.147 0.849 3.218
>
>
>
> Rx, Tx Checksum ON/Rx, Tx Vector OFF
> Test 64 Bytes 128 Bytes 256 Bytes 512 Bytes 1500 bytes Mix
> PHY-PHY-BIDIR 7.75 7.74 4.528 2.349 0.822 6.205
> PHY-VM-PHY-BIDIR 2.44 2.401 2.141 1.862 0.822 2.21
> PHY-VXLAN-PHY 3.53 3.53 3.53 2.147 0.849 3.52
>
> Findings:-
> 1) For bigger packet sizes, (<= 256 bytes) the performance numbers are almost the same in all the scenarios.
> 2) Tx checksum offload doesn’t give any performance improvement in all the case. In fact in some scenarios, it shows low performance compared to native.
> 3) Rx checksum offload provide tunneling performance improvement mainly on small sized packets.
>
> Please let me know your comments/thoughts on this?
Thanks, this is great data.
I agree that it doesn't make any sense to pursue transmit checksum
offload in OVS at this point since it doesn't result in improvements
in any case. I'm a bit surprised and it would be interesting to
understand in more detail what is going on and if anything can be
improved to allow both checksum offloading and vectoring at the same
time but I guess that's more of a DPDK question than one for OVS.
I'm a little bit torn as to whether we should apply your rx checksum
offload patch in the meantime while we wait for DPDK to offer the new
API. It looks like we'll have a 10% gain with tunneling in exchange
for a 1% loss in other situations, so the call obviously depends on
use case. Pravin, Daniele, others, any opinions?
Do you happen to know in the case of tunneling how the best case
(which looks like with rx checksums on) of outer UDP checksums
compares to tunneling without outer checksums (which would presumably
allow us to keep vectoring)? I should have mentioned this earlier, so
I'm more just curious if you have a ballpark figure.
More information about the dev
mailing list