[ovs-dev] [PATCH v1 0/9] conntrack: improve multithread scalability
Gaëtan Rivet
grive at u256.net
Wed Apr 21 22:16:53 UTC 2021
On Wed, Feb 17, 2021, at 17:34, Gaetan Rivet wrote:
> Conntracks are executed within the datapath. Locks along this path are crucial
> and their critical section should be minimal. The global 'ct_lock' is necessary
> before any action taken on connection states. This lock is needed for many
> operations on the conntrack, slowing down the datapath.
>
> The cleanup thread 'ct_clean' will take it to do its job. As it can hold it a
> long time, the thread is limited in amount of connection cleaned per round,
> and calls are rate-limited.
>
> * Timeout policies locking is contrived to avoid deadlock.
> Anytime a connection state is updated, during its update it is unlocked,
> 'ct_lock' is taken, then the connection is locked again. Then the reverse
> is done for unlock.
>
> * Scalability is poor. The global ct_lock needs to be taken before applying
> any change to a conn object. This is backward: local changes to smaller
> objects should be independent, then the global lock should only be taken once
> the rest of the work is done, the goal being to have the smallest possible
> critical section.
>
> It can be improved. Using RCU-friendly structures for connections, zone limits
> and timeout policies, read-first workload is improved and the precedence of the
> global 'ct_lock' and local 'conn->lock' can be inversed.
>
> Running the conntrack benchmark we see these changes:
> ./tests/ovstest test-conntrack benchmark <N> 3000000 32
>
> code \ N 1 2 4 8
> Before 2310 2766 6117 19838 (ms)
> After 2072 2084 2653 4541 (ms)
>
> One thread in the benchmark executes the task of a PMD, while the
> 'ct_clean' thread
> runs in background as well.
>
> Github actions: https://github.com/grivet/ovs/actions/runs/574446345
>
> Gaetan Rivet (9):
> conntrack: Use rcu-lists to store conn expirations
> conntrack: Use a cmap to store zone limits
> conntrack: Init hash basis first at creation
> conntrack-tp: Use a cmap to store timeout policies
> conntrack: Inverse conn and ct lock precedence
> conntrack: Do not schedule zero ms timers
> conntrack: Do not rate limit ct-sweep
> conntrack: Do not log empty ct-sweep
> conntrack: Use an atomic conn expiration value
>
> lib/conntrack-private.h | 96 +++++++++++++------
> lib/conntrack-tp.c | 137 +++++++++++++++-----------
> lib/conntrack.c | 206 +++++++++++++++++++++++++---------------
> lib/conntrack.h | 4 +-
> lib/dpif-netdev.c | 5 +-
> 5 files changed, 280 insertions(+), 168 deletions(-)
>
> --
> 2.30.0
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
The v2 of this series has been sent: https://patchwork.ozlabs.org/project/openvswitch/list/?series=240241
--
Gaetan Rivet
More information about the dev
mailing list