[ovs-dev] [PATCH v1 00/23] dpif-netdev: Parallel offload processing
grive at u256.net
Mon Mar 15 12:29:06 UTC 2021
On Mon, Mar 15, 2021, at 09:38, David Marchand wrote:
> On Wed, Feb 10, 2021 at 4:07 PM Gaetan Rivet <grive at u256.net> wrote:
> > This patch series aims to improve the performance of the management
> > of hw-offloads in dpif-netdev. In the current version, some setup
> > will experience high memory usage and poor latency between a flow
> > decision and its execution regarding hardware offloading.
> > This series starts by measuring key metrics regarding both issues
> > Those patches are introduced first to compare the current status
> > with each improvements introduced.
> > Offloads enqueued and inserted, as well as the latency
> > from queue insertion to hardware insertion is measured. A new
> > command 'ovs-appctl dpctl/offload-stats-show' is introduced
> > to show the current measure.
> > In my current performance test setup I am measuring an
> > average latency hovering between 1~2 seconds.
> > After the optimizations, it is reduced to 500~900 ms.
> > Finally when using multiple threads and with proper driver
> > support, it is measured in the order of 1 ms.
> > A few modules are introduced:
> > * An ID pool with reduced capabilities, simplifying its
> > operations and allowing better performances in both
> > single and multi-thread setup.
> > * A lockless queue between PMDs / revalidators and
> > offload thread(s). As the number of PMDs increases,
> > contention can be high on the shared queue.
> > This queue is designed to serve as message queue
> > between threads.
> > * A bounded lockless MPMC ring and some helpers for
> > calculating moving averages.
> > * A moving average module for Cumulative and Exponential
> > moving averages.
> > The netdev-offload-dpdk module is made thread-safe.
> > Internal maps are made per-netdev instead, and locks are
> > taken for shorter critical sections within the module.
> Hey Gaetan,
> This looks like an interesting series, I'll be looking too at it in
> the coming weeks.
> Just a first and easy update, I noticed an assert is triggered when
> stopping OVS (at least).
> assertion thread_is_hw_offload || thread_is_rcu failed in
> I did not check yet where the issue is, but I noticed it too half way
> of the series, and it was not on stop in this case.
> You can probably catch it easily.
> I have hw-offloads enabled, 2 pf ports, 2 representors ports in a
> single bridge, 2 pmds, no additional configuration.
> I usually have bi directional traffic running through OVS while I restart.
> David Marchand
Thanks for the report!
This abort is added in patch 12: netdev-offload: Add multi-thread API
It is relying on the offload thread being named 'hw_offload'.
This name is changed by patch 22: dpif-netdev: Use one or more offload threads
In the RFC series, a separate patch did a fix on the thread name .
I was relying on this fix happening first. I only did compilation checks in-between after squashing this patch.
I can either re-introduce the fix patch separately, or rewrite the check in patch 12, then update it in patch 22.
In my opinion having the fix separate was better but I can go with either solutions.
Do you have a preference?
(I did not have a setup this morning to do runtime checks, I will run them later. Reading the code this is my current understanding and it make sense for now.)
More information about the dev