[ovs-dev] [PATCH v1 00/23] dpif-netdev: Parallel offload processing

Gaëtan Rivet grive at u256.net
Mon Mar 15 12:29:06 UTC 2021


On Mon, Mar 15, 2021, at 09:38, David Marchand wrote:
> On Wed, Feb 10, 2021 at 4:07 PM Gaetan Rivet <grive at u256.net> wrote:
> >
> > This patch series aims to improve the performance of the management
> > of hw-offloads in dpif-netdev. In the current version, some setup
> > will experience high memory usage and poor latency between a flow
> > decision and its execution regarding hardware offloading.
> >
> > This series starts by measuring key metrics regarding both issues
> > Those patches are introduced first to compare the current status
> > with each improvements introduced.
> > Offloads enqueued and inserted, as well as the latency
> > from queue insertion to hardware insertion is measured. A new
> > command 'ovs-appctl dpctl/offload-stats-show' is introduced
> > to show the current measure.
> >
> > In my current performance test setup I am measuring an
> > average latency hovering between 1~2 seconds.
> > After the optimizations, it is reduced to 500~900 ms.
> > Finally when using multiple threads and with proper driver
> > support[1], it is measured in the order of 1 ms.
> >
> > A few modules are introduced:
> >
> >   * An ID pool with reduced capabilities, simplifying its
> >     operations and allowing better performances in both
> >     single and multi-thread setup.
> >
> >   * A lockless queue between PMDs / revalidators and
> >     offload thread(s). As the number of PMDs increases,
> >     contention can be high on the shared queue.
> >     This queue is designed to serve as message queue
> >     between threads.
> >
> >   * A bounded lockless MPMC ring and some helpers for
> >     calculating moving averages.
> >
> >   * A moving average module for Cumulative and Exponential
> >     moving averages.
> >
> > The netdev-offload-dpdk module is made thread-safe.
> > Internal maps are made per-netdev instead, and locks are
> > taken for shorter critical sections within the module.
> 
> Hey Gaetan,
> 
> This looks like an interesting series, I'll be looking too at it in
> the coming weeks.
> Just a first and easy update, I noticed an assert is triggered when
> stopping OVS (at least).
> 2021-03-11T20:32:01.928Z|00350|util|EMER|lib/netdev-offload.c:479:
> assertion thread_is_hw_offload || thread_is_rcu failed in
> netdev_offload_thread_init()
> 
> I did not check yet where the issue is, but I noticed it too half way
> of the series, and it was not on stop in this case.
> You can probably catch it easily.
> 
> I have hw-offloads enabled, 2 pf ports, 2 representors ports in a
> single bridge, 2 pmds, no additional configuration.
> I usually have bi directional traffic running through OVS while I restart.
> 
> 
> -- 
> David Marchand

Hey David,

Thanks for the report!
This abort is added in patch 12: netdev-offload: Add multi-thread API
It is relying on the offload thread being named 'hw_offload'.
This name is changed by patch 22: dpif-netdev: Use one or more offload threads

In the RFC series, a separate patch did a fix on the thread name [1].
I was relying on this fix happening first. I only did compilation checks in-between after squashing this patch.

I can either re-introduce the fix patch separately, or rewrite the check in patch 12, then update it in patch 22.
In my opinion having the fix separate was better but I can go with either solutions.

Do you have a preference?

(I did not have a setup this morning to do runtime checks, I will run them later. Reading the code this is my current understanding and it make sense for now.)

[1]: https://mail.openvswitch.org/pipermail/ovs-dev/2020-December/378337.html
-- 
Gaetan Rivet


More information about the dev mailing list