[ovs-dev] thundering herd wakeup of handler threads
dsahern at gmail.com
Tue Dec 10 19:13:27 UTC 2019
On a hypervisor running a 4.14.91 kernel and OVS 2.11 I am seeing a
thundering herd wake up problem. Every packet punted to userspace wakes
up every one of the handler threads. On a box with 96 cpus, there are 71
handler threads which means 71 process wakeups for every packet punted.
This is really easy to see, just watch sched:sched_wakeup tracepoints.
With a few extra probes:
perf probe sock_def_readable sk=%di
perf probe ep_poll_callback wait=%di mode=%si sync=%dx key=%cx
perf probe __wake_up_common wq_head=%di mode=%si nr_exclusive=%dx
you can see there is a single netlink socket and its wait queue contains
an entry for every handler thread.
This does not happen with the 2.7.3 version. Roaming commits it appears
that the change in behavior comes from this commit:
Author: Matteo Croce <mcroce at redhat.com>
Date: Tue Sep 25 10:51:05 2018 +0200
dpif-netlink: don't allocate per thread netlink sockets
Is this a known problem?
More information about the dev