[ovs-dev] RFC: A plan for stats.

[Daniele Di Proietto]

Hi Ethan,

As you know I¹ve been working on similar problems for the userspace
datapath. Also, I¹m about to post a patch series to reduce the overhead of
counting (global, not per-flow) stats in dpif-netdev.c.
Here are some thoughts :

* I definitely think it¹s a good idea to use ovsthread_stats for
rule_dpif_stats, thanks for working on that.
* It makes sense to improve ovsthread_stats interface like you suggested.
I can spend some time on that, if you think it¹s worth.

* ovsthread_stats just uses an approximation of thread local storage.
Basically it is not convenient to use truly per-thread data because:
  * The availability of per thread keys might be limited. This, of course,
can be worked around, but I don¹t know about performance.
  * Most importantly, many implementation of thread local storage (e.g.
C11) do not allow cross-thread access (there¹s a big comment in
lib/ovs-thread.h).

* For the userspace datapath we can have per thread statistics, because we
stricly control the threads that have access to the datapath (pmd
pthreads, plus non pmd access are already protected by a mutex). My patch
should take care of datapath statistics, and once we have per pmd thread
classifier we will have per pmd thread flows that do not need statistics.

* Since ovsthread_stats doesn¹t use per thread data, I don¹t know if it
will be possible to remove the mutex and use the trick you described.
Maybe I can use that trick in my patch for 32-bit systems (let¹s discuss
this later)

What do you guys think?

Daniele

On 9/20/14, 11:00 AM, "Ethan Jackson" <ethan at nicira.com> wrote:

>I'm doing some performance testing which heavily stresses the OVS DPDK
>slow
>path, and notice we spend a ridiculous amount of CPU time on the
>rule_dpif stats
>mutex.  For my test case, by commenting out the mutex I literally doubled
>the
>performance.  I'm in a bit of a rush, so as a short term hack, I simply
>copied
>over the per-thread stats code which dpif-netdev uses, but it got me
>thinking of
>a general plan for how we should handle stats in userspace which I'll
>outline
>here:
>
>To start, I think the ovsthread_stats construction needs to be pulled
>into it's
>own library and made higher level.  Right now it's a general mechanism for
>storing arbitrary per thread data.  I think we really want something which
>implements an API something like "get_stats", "credit_stats",
>"clear_stats",
>etc.  That could be used directly by both ofproto-dpif and dpif-netdev
>very
>cleanly.  You'll see in the hacked together patch I've included below,
>it's
>basically a direct copy of all the relevant dpif-netdev code.
>
>That done, I think we can actually get rid of the flow stats mutex by
>using a
>technique similar to what we do for cmaps.  We're guaranteed that only one
>writer ever touches a particular bucket, so it should be sufficient to
>ensure
>all of the data fits on a particular cache line, and that we update a
>sequence
>number to notify readers of changes.
>
>Thoughts?
>
>Ethan