[ovs-discuss] ovs-vswitchd mlockall and stack size

[Ben Pfaff]

I ran that command on my own development host rather than in the Xen
DDK.  (I use i386 instead of x86-64 to put some diversity into
development, since most developers use x86-64.)  Rest assured that we
do all of our XenServer builds in the Xen DDK that corresponds to the
target XenServer target (and we always have don so).

32- versus 64-bit shouldn't affect where the best opportunities for
reducing stack use lie.

On Wed, Jul 09, 2014 at 03:12:14PM +0100, Anoob Soman wrote:
> Hi Ben,
> 
> From the objdump-sed (%esp) command I realised that you guys might
> be using a XenServer DDK shipped with 6.2, which is 32bit. From
> Creedence (next release) onwards dom0 is 64bit, and so are the
> tools. http://xenserver.org/open-source-virtualization-download/24-product/creedence/143-xs-2014-development-snapshots.html
> 
> Thanks,
> Anoob.
> On 08/07/14 17:47, Ben Pfaff wrote:
> >I guess that the biggest effect on stack size would be the flow table
> >and in particular how much recursion flow processing causes.  There are
> >a few tests that force as-deep-as-possible recursion:
> >
> >     AT_SETUP([ofproto-dpif - infinite resubmit])
> >     AT_SETUP([ofproto-dpif - exponential resubmit chain])
> >
> >I don't think that forcing all packets to userspace would have much of
> >an effect.  (The closest equivalent would be to disable megaflows,
> >there's an "ovs-appctl" command for that, look in "ovs-appctl help".)
> >
> >Another hint toward maximum stack requirement is to look through the
> >generated asm for stack usage, e.g.:
> >
> >         objdump -dr vswitchd/ovs-vswitchd|sed -n 's/^.*sub.*$0x\([0-9a-f]\{1,\}\),%esp/\1/p'|sort|uniq|less
> >
> >which shows that we have at least one place where we allocate 327,788
> >bytes on the stack (!).  I hope that is not in the flow processing path!
> >
> >On Tue, Jul 08, 2014 at 05:36:07PM +0100, Anoob Soman wrote:
> >>I have been running tests with 1MB stack size and ovs-vswitchd seem
> >>to hold pretty well. I will try to do some more experiments to find
> >>out the max depth of the stack, but I am afraid this will totally
> >>depend on the test I am running. Any suggestion on what sort of test
> >>I should be running ? More over "force-miss-model" other-config is
> >>missing from 2.1.x as there is no concept of facets. Is there way
> >>that I can force all packets to be processed in userspace, other
> >>than me doing "ovs-dpctl del-flows" periodically.
> >>
> >>Thanks,
> >>Anoob.
> >>On 08/07/14 17:15, Ben Pfaff wrote:
> >>>On Tue, Jul 08, 2014 at 05:08:43PM +0100, Anoob Soman wrote:
> >>>>Since openvswitch has moved to multi-threaded model, RSS usage of
> >>>>ovs-vswitchd has increased quite significantly compared to the last
> >>>>release we used (ovs-1.4.x). Part of the problem is using mlockall
> >>>>(with MCL_CURRENT|MCL_FUTURE) on ovs-vswitchd, which causes every
> >>>>pthreads stack's and heap's virtual address to locked to RAM.
> >>>>ovs-vswitch (2.1.x) running on a 8 vCPU dom0 (10 pthreads) uses
> >>>>around 89M of RSS (80MB just for stack), without any VMs running on
> >>>>the host. One way to reduce RSS would be to reduce the number of
> >>>>"n-handler-threads" and "n-revalidator-threads", but I am not sure
> >>>>about the performance impact of having these thread numbers reduced.
> >>>>I am wondering if the stack size of the pthreads can be reduce
> >>>>(using pthread_attr_setstack). By default pthreads max stack size is
> >>>>8MB and mlockall locks all of this 8MB into RAM. What could be
> >>>>optimal stack size that can be used.
> >>>I think it would be very reasonable to reduce the stack sizes, but I
> >>>don't know the "correct" size off-hand.  Since you're looking at the
> >>>problem already, perhaps you should consider some experiments.
>