[ovs-dev] [PATCH 0/3] Output packet batching.

[Jan Scheurich]

I like this generic approach to collect the packets to be output per port for each Rx batch in dpif-netdev. It is indeed simpler than the approach in [1].

However, [1] originally had a larger scope, namely to buffer packets in an intermediate queue per netdev tx queue *across* multiple rx batches. Unfortunately this aspect was "optimized" away in v3 to minimize the impact on minimum and average latency for ports with little traffic.

Limiting the output batching to a single rx batch has little benefit unless the average rx batch size is significant. According to our measurements with instrumented netdev-dpdk datapath this happens only very close to or above PMD saturation. At realistic production load levels (say 95% PMD processing cycles) the average rx batch size is still around 1-2 only, so the gain we can expect is negligible. The primary use case for this patch appears to be to push up the saturation throughput in benchmarks.

In our perspective a perhaps more important use case for tx batching would be to reduce the number of virtio interrupts a PMD needs to trigger in Qemu/KVM guests using the kernel virtio-net driver (currently one per tx batch).

In internal benchmarks with kernel applications we have observed that the OVS PMD spends up to 30% of its cycles kicking the eventfd and the guest OS in the VM spends 50% or more of a vCPU processing virtio-net interrupts. This seriously degrades the performance of both OVS and application.

With a vhostuser tx batching patch that is not limited to a single Rx batch but relies on periodic flushing (e.g. every 50-100 us), we have been able to reduce this interrupt overhead and significantly improve e.g. the iperf performance.

My suggestion would be to complement the generic output packet batching per rx batch in dpif-netdev in this patch with specific support for tx batching for vhostuser ports in netdev-dpdk, using an intermediate queue with a configurable flushing interval (including the possibility to turn buffering off when latency is an issue).

BR, Jan

> -----Original Message-----
> From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-bounces at openvswitch.org] On Behalf Of Ilya Maximets
> Sent: Friday, 30 June, 2017 14:03
> To: ovs-dev at openvswitch.org; Bhanuprakash Bodireddy <bhanuprakash.bodireddy at intel.com>
> Cc: Heetae Ahn <heetae82.ahn at samsung.com>; Ilya Maximets <i.maximets at samsung.com>
> Subject: [ovs-dev] [PATCH 0/3] Output packet batching.
> 
> This patch-set inspired by [1] from Bhanuprakash Bodireddy.
> Implementation of [1] looks very complex and introduces many pitfalls for
> later code modifications like possible packet stucks.
> 
> This version targeted to make simple and flexible output packet batching on
> higher level without introducing and even simplifying netdev layer.
> 
> Patch set consists of 3 patches. All the functionality introduced in the
> first patch. Two others are just cleanups of netdevs to not do unnecessary
> things.
> 
> Basic testing of 'PVP with OVS bonding on phy ports' scenario shows
> significant performance improvement.
> More accurate and intensive testing required.
> 
> [1] [PATCH 0/6] netdev-dpdk: Use intermediate queue during packet transmission.
>     https://mail.openvswitch.org/pipermail/ovs-dev/2017-June/334762.html
> 
> Ilya Maximets (3):
>   dpif-netdev: Output packet batching.
>   netdev: Remove unused may_steal.
>   netdev: Remove useless cutlen.
> 
>  lib/dpif-netdev.c     | 81 ++++++++++++++++++++++++++++++++++++---------------
>  lib/netdev-bsd.c      |  7 ++---
>  lib/netdev-dpdk.c     | 30 +++++++------------
>  lib/netdev-dummy.c    |  6 ++--
>  lib/netdev-linux.c    |  7 ++---
>  lib/netdev-provider.h |  7 ++---
>  lib/netdev.c          | 12 +++-----
>  lib/netdev.h          |  2 +-
>  8 files changed, 83 insertions(+), 69 deletions(-)
> 
> --
> 2.7.4
> 
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev