[ovs-dev] userspace datapath performance (was: Re: Threaded userspace datapath)
rizzo at iet.unipi.it
Wed Oct 10 23:13:44 UTC 2012
On Wed, Oct 10, 2012 at 09:20:19AM -0700, Ben Pfaff wrote:
> More than anything I'd like to see an analysis, based on at least
> simple measurements, probably including some conjecture, of the reason
> or reasons that the threaded datapath yields performance that is so
> much better. If it comes down to something not fundamental to having
> threads, then we can talk about the trade-offs between working on
> those issues directly versus introducing threading.
> One issue that's been brought up is the fact that Open vSwitch throws
> away the information provided by "poll" regarding the file descriptors
> that are ready for read or write. If that's a significant cost that
> contributes to slowness (I'm not convinced yet, but I could be
> convinced by measurements), then that's something that we can work on;
For measurements see our infocom paper at
(a preliminary version and other data are available on the netmap page).
We saw 50Kpps tops with the single event loop, and about 300Kpps
with split threads, in both cases processing one packet per iteration,
and using bpf (FreeBSD) or PF_PACKET (?) on linux.
Increasing the number of packets per iteration approx. doubles the
throughput for split threads (the bottleneck here is the system
call to do the packet I/O), whereas gives a negligible improvement
in the single event loop case (suggesting that there is a ton of
work done at each iteration).
> I'm happy to do that. I think that we can work on that without
> completely changing how the poll loop works.
I am sure you can get some speedup with some relatively simple
optimizations, I am just unclear by how much (increasing the batch
size, which was another problem, did not help in the single-thread
Another problem, which i suspect is harder to fix, is that the code
before select()/poll() to determine which descriptors we should
look at has a lot more work to do than simply marking the descriptors
associated to open interfaces/tunnels.
Not to mention that if the single thread is busy communicating with
the controller, logging, resolving names or whatnot, latency and
throughput become extremely hard to control.
I do buy the concerns in using a second thread as opposed to a
process. However, using a thread at least avoids having to pass
around information (updates to the forwarding table, file descriptors
for interfaces, etc.) which would require additional complications.
True, this comes at the price of adding some locking around data
structures, but updates to these data structures should be relatively
infrequent and so even a crude locking scheme should work well and
be reasonably easy to audit.
And, to be perfectly frank, we do not have neither time nor motivation
to try and accelerate a single threaded version, which we believe
is inherently limited for the reasons mentioned above.
We would rather put some work on integrate the openflow forwarding
code within our VALE switch
which is based on netmap and works on both FreeBSD and Linux,
and could be used to speedup the in-kernel forwarding.
Anyways, let's see if we can find some agreement on how to
proceed with a face-to-face meeting
More information about the dev