[ovs-discuss] Performance counters for vswitchd

Ben Pfaff blp at nicira.com
Mon Jul 7 17:44:32 UTC 2014


On Mon, Jul 07, 2014 at 08:48:52PM +0400, Andrey Korolyov wrote:
> On Mon, Jul 7, 2014 at 8:36 PM, Ben Pfaff <blp at nicira.com> wrote:
> > On Mon, Jul 07, 2014 at 08:33:50PM +0400, Andrey Korolyov wrote:
> >> On Mon, Jul 7, 2014 at 7:15 PM, Ben Pfaff <blp at nicira.com> wrote:
> >> > On Fri, Jul 04, 2014 at 09:02:40PM +0400, Andrey Korolyov wrote:
> >> >> On Fri, Jul 4, 2014 at 8:39 PM, Ben Pfaff <blp at nicira.com> wrote:
> >> >> > On Fri, Jul 04, 2014 at 02:39:18PM +0400, Andrey Korolyov wrote:
> >> >> >> is there any plans around to introduce built-in performance statistics
> >> >> >> for userspace switch process? Right now one is able to check memory
> >> >> >> consumption statistics and it will be quite awesome to introduce kind
> >> >> >> of interface for measurement of thread performance too.
> >> >> >
> >> >> > We have performance counters.  What do you want to measure?
> >> >>
> >> >> For example, total cpu utilization by workers not involving external
> >> >> monitoring tools. There are two cases - reactive flows, which
> >> >> currently have almost linear characteristic for utilization vs flow
> >> >> rate when adding unique flows on moderate speed (30k/s with six cores
> >> >> of E5 series as a ceiling), and proactive mask/asterisk flows which
> >> >> has sublinear characteristic even being far away from the actual
> >> >> ceiling. In the second case vswitchd heavily relies on decisions made
> >> >> inside module` heuristic, so same flow hit rate can produce very
> >> >> different cpu utilization for userspace vswitchd.
> >> >
> >> > I'm not sure how OVS would do a better job of that than external
> >> > monitoring tools.  Doing a good job would probably mean producing a time
> >> > series for each thread's CPU usage.  A regular log file isn't ideal for
> >> > that.
> >>
> >> Ok, I see. Jiffie counting is a pidstat job, but I thought that there
> >> is a easier way around.
> >
> > Are you talking about performance monitoring for work done in the kernel
> > module (e.g. from bottom halves etc.)?  I'm only talking about
> > userspace.  I don't know the relevant kernel performance monitoring
> > infrastructure and can't speak for the right way to do it.
> 
> No, just userspace. Bh profiling looks unnecessary since 2.1+ module
> doing very great job as a packet processor. I was afraid that the
> unmonitored vswitchd process may eat significant part of the compute
> node and may get unnoticed with couple of conditions, e.g. relatively
> small packet rate with high flow rate or high hit rate on the reactive
> flows (if controller is stupid enough to just report the rate, not to
> analyze/group asking behavior). Of course, most straightforward way to
> prevent this is to check process consumption via pidstat or another
> top-like jiffie counter, I just thought of some ready-to-use mechanism
> inside vswitchd itself which I was not simply aware of.

OVS has a mechanism (for years) that's designed to point out code that
causes 100% consumption: if a thread starts using more than 50% of a
CPU, then it starts logging its own CPU consumption and the line of code
that caused the wakeup.  I don't know whether that is good enough for
your needs.



More information about the discuss mailing list