[ovs-dev] [PATCH v1 0/9] conntrack: improve multithread scalability

Gaëtan Rivet grive at u256.net
Wed Apr 21 22:16:53 UTC 2021


On Wed, Feb 17, 2021, at 17:34, Gaetan Rivet wrote:
> Conntracks are executed within the datapath. Locks along this path are crucial
> and their critical section should be minimal. The global 'ct_lock' is necessary
> before any action taken on connection states. This lock is needed for many
> operations on the conntrack, slowing down the datapath.
> 
> The cleanup thread 'ct_clean' will take it to do its job. As it can hold it a
> long time, the thread is limited in amount of connection cleaned per round,
> and calls are rate-limited.
> 
> * Timeout policies locking is contrived to avoid deadlock.
>   Anytime a connection state is updated, during its update it is unlocked,
>   'ct_lock' is taken, then the connection is locked again. Then the reverse
>   is done for unlock.
> 
> * Scalability is poor. The global ct_lock needs to be taken before applying
>   any change to a conn object. This is backward: local changes to smaller
>   objects should be independent, then the global lock should only be taken once
>   the rest of the work is done, the goal being to have the smallest possible
>   critical section.
> 
> It can be improved. Using RCU-friendly structures for connections, zone limits
> and timeout policies, read-first workload is improved and the precedence of the
> global 'ct_lock' and local 'conn->lock' can be inversed.
> 
> Running the conntrack benchmark we see these changes:
>   ./tests/ovstest test-conntrack benchmark <N> 3000000 32
> 
> code \ N      1     2     4     8
>   Before   2310  2766  6117 19838  (ms)
>    After   2072  2084  2653  4541  (ms)
> 
> One thread in the benchmark executes the task of a PMD, while the 
> 'ct_clean' thread
> runs in background as well.
> 
> Github actions: https://github.com/grivet/ovs/actions/runs/574446345
> 
> Gaetan Rivet (9):
>   conntrack: Use rcu-lists to store conn expirations
>   conntrack: Use a cmap to store zone limits
>   conntrack: Init hash basis first at creation
>   conntrack-tp: Use a cmap to store timeout policies
>   conntrack: Inverse conn and ct lock precedence
>   conntrack: Do not schedule zero ms timers
>   conntrack: Do not rate limit ct-sweep
>   conntrack: Do not log empty ct-sweep
>   conntrack: Use an atomic conn expiration value
> 
>  lib/conntrack-private.h |  96 +++++++++++++------
>  lib/conntrack-tp.c      | 137 +++++++++++++++-----------
>  lib/conntrack.c         | 206 +++++++++++++++++++++++++---------------
>  lib/conntrack.h         |   4 +-
>  lib/dpif-netdev.c       |   5 +-
>  5 files changed, 280 insertions(+), 168 deletions(-)
> 
> --
> 2.30.0
> 
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> 

The v2 of this series has been sent: https://patchwork.ozlabs.org/project/openvswitch/list/?series=240241

-- 
Gaetan Rivet


More information about the dev mailing list