[ovs-discuss] Performance counters for vswitchd

Andrey Korolyov andrey at xdel.ru
Mon Jul 7 16:48:52 UTC 2014


On Mon, Jul 7, 2014 at 8:36 PM, Ben Pfaff <blp at nicira.com> wrote:
> On Mon, Jul 07, 2014 at 08:33:50PM +0400, Andrey Korolyov wrote:
>> On Mon, Jul 7, 2014 at 7:15 PM, Ben Pfaff <blp at nicira.com> wrote:
>> > On Fri, Jul 04, 2014 at 09:02:40PM +0400, Andrey Korolyov wrote:
>> >> On Fri, Jul 4, 2014 at 8:39 PM, Ben Pfaff <blp at nicira.com> wrote:
>> >> > On Fri, Jul 04, 2014 at 02:39:18PM +0400, Andrey Korolyov wrote:
>> >> >> is there any plans around to introduce built-in performance statistics
>> >> >> for userspace switch process? Right now one is able to check memory
>> >> >> consumption statistics and it will be quite awesome to introduce kind
>> >> >> of interface for measurement of thread performance too.
>> >> >
>> >> > We have performance counters.  What do you want to measure?
>> >>
>> >> For example, total cpu utilization by workers not involving external
>> >> monitoring tools. There are two cases - reactive flows, which
>> >> currently have almost linear characteristic for utilization vs flow
>> >> rate when adding unique flows on moderate speed (30k/s with six cores
>> >> of E5 series as a ceiling), and proactive mask/asterisk flows which
>> >> has sublinear characteristic even being far away from the actual
>> >> ceiling. In the second case vswitchd heavily relies on decisions made
>> >> inside module` heuristic, so same flow hit rate can produce very
>> >> different cpu utilization for userspace vswitchd.
>> >
>> > I'm not sure how OVS would do a better job of that than external
>> > monitoring tools.  Doing a good job would probably mean producing a time
>> > series for each thread's CPU usage.  A regular log file isn't ideal for
>> > that.
>>
>> Ok, I see. Jiffie counting is a pidstat job, but I thought that there
>> is a easier way around.
>
> Are you talking about performance monitoring for work done in the kernel
> module (e.g. from bottom halves etc.)?  I'm only talking about
> userspace.  I don't know the relevant kernel performance monitoring
> infrastructure and can't speak for the right way to do it.

No, just userspace. Bh profiling looks unnecessary since 2.1+ module
doing very great job as a packet processor. I was afraid that the
unmonitored vswitchd process may eat significant part of the compute
node and may get unnoticed with couple of conditions, e.g. relatively
small packet rate with high flow rate or high hit rate on the reactive
flows (if controller is stupid enough to just report the rate, not to
analyze/group asking behavior). Of course, most straightforward way to
prevent this is to check process consumption via pidstat or another
top-like jiffie counter, I just thought of some ready-to-use mechanism
inside vswitchd itself which I was not simply aware of.



More information about the discuss mailing list