[ovs-dev] [PATCH v1 12/23] netdev-offload: Add multi-thread API

Gaëtan Rivet grive at u256.net
Mon Mar 8 17:21:34 UTC 2021


On Mon, Mar 8, 2021, at 16:04, Maxime Coquelin wrote:
> Hi Gaetan,
> 
> On 2/10/21 4:33 PM, Gaetan Rivet wrote:
> > Expose functions reporting user configuration of offloading threads, as
> > well as utility functions for multithreading.
> > 
> > This will only expose the configuration knob to the user, while no
> > datapath will implement the multiple thread request.
> > 
> > This will allow implementations to use this API for offload thread
> > management in relevant layers before enabling the actual dataplane
> > implementation.
> > 
> > The offload thread ID is lazily allocated and can as such be in a
> > different order than the offload thread start sequence.
> > 
> > The RCU thread will sometime access hardware-offload objects from
> > a provider for reclamation purposes.  In such case, it will get
> > a default offload thread ID of 0. Care must be taken that using
> > this thread ID is safe concurrently with the offload threads.
> > 
> > Signed-off-by: Gaetan Rivet <grive at u256.net>
> > Reviewed-by: Eli Britstein <elibr at nvidia.com>
> > ---
> >  lib/netdev-offload-provider.h |  1 +
> >  lib/netdev-offload.c          | 88 ++++++++++++++++++++++++++++++++++-
> >  lib/netdev-offload.h          | 19 ++++++++
> >  vswitchd/vswitch.xml          | 16 +++++++
> >  4 files changed, 122 insertions(+), 2 deletions(-)
> > 
> > diff --git a/lib/netdev-offload-provider.h b/lib/netdev-offload-provider.h
> > index 4815a1bd2..f8b66961d 100644
> > --- a/lib/netdev-offload-provider.h
> > +++ b/lib/netdev-offload-provider.h
> > @@ -84,6 +84,7 @@ struct netdev_flow_api {
> >                      struct dpif_flow_stats *);
> >  
> >      /* Get the number of flows offloaded to netdev.
> > +     * 'n_flows' is an array of counters, one per offload thread.
> >       * Return 0 if successful, otherwise returns a positive errno value. */
> >      int (*flow_get_n_flows)(struct netdev *, uint64_t *n_flows);
> >  
> > diff --git a/lib/netdev-offload.c b/lib/netdev-offload.c
> > index 3092f8d55..c30377b91 100644
> > --- a/lib/netdev-offload.c
> > +++ b/lib/netdev-offload.c
> > @@ -60,6 +60,12 @@ VLOG_DEFINE_THIS_MODULE(netdev_offload);
> >  
> >  static bool netdev_flow_api_enabled = false;
> >  
> > +#define DEFAULT_OFFLOAD_THREAD_NB 1
> > +#define MAX_OFFLOAD_THREAD_NB 10
> 
> Out of curiosity, is there a rationale in setting the maximum number of
> threads to 10?
> 
> Maxime

Hi Maxime,

The limit is mostly arbitrary. I didn't want to allow setting outlandish values, some memory allocations depends on it.
In our driver, scaling is effective up to 4 threads at most, and one core use-case of OVS for us, Bluefield, has 8 cores.

10 threads seemed a reasonable limit. I think it's good to have a margin, to test with higher-than-effective numbers in development, while in production we expect on our end to run 4 threads in general.

Maybe the limit could be removed, similar to n-handle-threads and n-revalidator-threads.
On that point: those two will adapt to each other and depend on the number of CPU cores. Maybe we might also want to allow some budget for offload thread(s), and possibly ct_clean as well.

-- 
Gaetan Rivet


More information about the dev mailing list