[ovs-dev] [PATCH v1 00/23] dpif-netdev: Parallel offload processing
Gaetan Rivet
grive at u256.net
Wed Feb 10 14:57:18 UTC 2021
This patch series aims to improve the performance of the management
of hw-offloads in dpif-netdev. In the current version, some setup
will experience high memory usage and poor latency between a flow
decision and its execution regarding hardware offloading.
This series starts by measuring key metrics regarding both issues
Those patches are introduced first to compare the current status
with each improvements introduced.
Offloads enqueued and inserted, as well as the latency
from queue insertion to hardware insertion is measured. A new
command 'ovs-appctl dpctl/offload-stats-show' is introduced
to show the current measure.
In my current performance test setup I am measuring an
average latency hovering between 1~2 seconds.
After the optimizations, it is reduced to 500~900 ms.
Finally when using multiple threads and with proper driver
support[1], it is measured in the order of 1 ms.
A few modules are introduced:
* An ID pool with reduced capabilities, simplifying its
operations and allowing better performances in both
single and multi-thread setup.
* A lockless queue between PMDs / revalidators and
offload thread(s). As the number of PMDs increases,
contention can be high on the shared queue.
This queue is designed to serve as message queue
between threads.
* A bounded lockless MPMC ring and some helpers for
calculating moving averages.
* A moving average module for Cumulative and Exponential
moving averages.
The netdev-offload-dpdk module is made thread-safe.
Internal maps are made per-netdev instead, and locks are
taken for shorter critical sections within the module.
CI result: https://github.com/grivet/ovs/actions/runs/554918929
[1]: The rte_flow API was made thread-safe in the 20.11 DPDK
release. Drivers that do not implement those operations
concurrently are protected by a lock. Others will
allow better concurrency, that improve the result
of this series.
Gaetan Rivet (23):
netdev: Add flow API de-init function
netdev-offload-dpdk: Use per-netdev offload metadata
netdev-offload-dpdk: Implement hw-offload statistics read
dpctl: Add function to read hardware offload statistics
dpif-netdev: Rename offload thread structure
mov-avg: Add a moving average helper structure
dpif-netdev: Implement hardware offloads stats query
ovs-atomic: Expose atomic exchange operation
mpsc-queue: Module for lock-free message passing
llring: Add lockless MPMC bounded queue structure
seq-pool: Module for faster ID generation
netdev-offload: Add multi-thread API
dpif-netdev: Quiesce offload thread periodically
dpif-netdev: Postpone flow offload item freeing
dpif-netdev: Use seq-pool for mark allocation
netdev-offload-dpdk: Use per-thread HW offload stats
netdev-offload-dpdk: Lock rte_flow map access
netdev-offload-dpdk: Protect concurrent offload destroy/query
dpif-netdev: Use lockless queue to manage offloads
dpif-netdev: Make megaflow and mark mappings thread objects
dpif-netdev: Replace port mutex by rwlock
dpif-netdev: Use one or more offload threads
netdev-dpdk: Remove rte-flow API access locks
lib/automake.mk | 7 +
lib/dpctl.c | 36 +++
lib/dpif-netdev.c | 506 ++++++++++++++++++++---------
lib/dpif-netlink.c | 1 +
lib/dpif-provider.h | 7 +
lib/dpif.c | 8 +
lib/dpif.h | 9 +
lib/llring.c | 153 +++++++++
lib/llring.h | 76 +++++
lib/mov-avg.h | 171 ++++++++++
lib/mpsc-queue.c | 190 +++++++++++
lib/mpsc-queue.h | 149 +++++++++
lib/netdev-dpdk.c | 6 -
lib/netdev-offload-dpdk.c | 270 ++++++++++++++--
lib/netdev-offload-provider.h | 4 +
lib/netdev-offload.c | 92 +++++-
lib/netdev-offload.h | 21 ++
lib/ovs-atomic-c++.h | 3 +
lib/ovs-atomic-clang.h | 5 +
lib/ovs-atomic-gcc4+.h | 5 +
lib/ovs-atomic-gcc4.7+.h | 5 +
lib/ovs-atomic-i586.h | 5 +
lib/ovs-atomic-locked.h | 9 +
lib/ovs-atomic-msvc.h | 22 ++
lib/ovs-atomic-pthreads.h | 5 +
lib/ovs-atomic-x86_64.h | 5 +
lib/ovs-atomic.h | 8 +-
lib/seq-pool.c | 198 ++++++++++++
lib/seq-pool.h | 66 ++++
tests/automake.mk | 2 +
tests/library.at | 10 +
tests/test-mpsc-queue.c | 580 ++++++++++++++++++++++++++++++++++
tests/test-seq-pool.c | 543 +++++++++++++++++++++++++++++++
vswitchd/vswitch.xml | 16 +
34 files changed, 3007 insertions(+), 186 deletions(-)
create mode 100644 lib/llring.c
create mode 100644 lib/llring.h
create mode 100644 lib/mov-avg.h
create mode 100644 lib/mpsc-queue.c
create mode 100644 lib/mpsc-queue.h
create mode 100644 lib/seq-pool.c
create mode 100644 lib/seq-pool.h
create mode 100644 tests/test-mpsc-queue.c
create mode 100644 tests/test-seq-pool.c
--
2.30.0
More information about the dev
mailing list