[ovs-discuss] ovs-vswitchd mlockall and stack size

[Anoob Soman]

Hi Ben,

Thanks for the suggestions. I did some quick test and analysis of the 
stack usage on ovs-2.1.2 (planning to do it on master later) and here 
are some of the findings.

The list shows the stack usage (in bytes) by each of the functions. I 
have collected only those functions which uses more than 1KB of stack.
dpif_linux_operate             114536
updif_upcall_handler            70856
nl_sock_recv__                  65656
json_from_stream                8216
udpif_revalidator               7672
revalidator_sweep__             6008
system_stats_thread_func        4872
netdev_linux_run                4408
dpif_linux_port_poll            4328
nln_run                         4208
nl_sock_transact_multiple       3368
xlate_actions                   3112
ofproto_trace                   2760
handle_openflow__               2072
do_xlate_actions                1928
handle_flow_stats_request       1784
ofp_print_nxst_flow_monitor_reply 1712
handle_aggregate_stats_request  1368
netdev_linux_sys_get_stats      1352
append_group_stats &
ofproto_dpif_execute_actions    1320
parse_odp_key_mask_attr         1304
xlate_group_bucket              1272
dpif_ipfix_cache_expire
& describe_fd                   1256
dpif_linux_execute              1160
handle_flow_monitor_request     1112
sfl_agent_sysError              1080
handle_meter_mod                1048
sfl_agent_error                 1032

As you can see there are few functions that are in packet processing path.

Assuming that we ran a "AT_SETUP([ofproto-dpif - infinite resubmit])" 
test, which cause do_xlate_actions (and friends) to re-curse (64 levels 
deep), roughly around 400KB of stack would be used by 
"updif_upcall_handler". I think the stack usage of udpif_revalidator 
should be same as that of updif_upcall_handler (if not less). I limited 
the stack size of all the pthreads to 512KB and was able to run both the 
tests you mentioned.

I tried valgrind (tool=massif) against ovs-vswitchd and ran 
"AT_SETUP([ofproto-dpif - infinite resubmit])" test and valgrind 
reported the max stack usage was around 400MB and
"AT_SETUP([ofproto-dpif - exponential resubmit chain])" uses around 
700MB. This was with 4vCPUs (6 pthreads). valgrind though reports the 
total stack usage by all threads.

This makes me believe that 1MB of stack size should be enough for each 
of pthreads and 512KB would be tight. Let me know your thoughts. I will 
send out a patch which would limit pthread stack size to 1024KB and 
would it make it "other-config" configurable.

Thanks,
Anoob.

On 08/07/14 17:47, Ben Pfaff wrote:
> I guess that the biggest effect on stack size would be the flow table
> and in particular how much recursion flow processing causes.  There are
> a few tests that force as-deep-as-possible recursion:
>
>      AT_SETUP([ofproto-dpif - infinite resubmit])
>      
>
> I don't think that forcing all packets to userspace would have much of
> an effect.  (The closest equivalent would be to disable megaflows,
> there's an "ovs-appctl" command for that, look in "ovs-appctl help".)
>
> Another hint toward maximum stack requirement is to look through the
> generated asm for stack usage, e.g.:
>
>          objdump -dr vswitchd/ovs-vswitchd|sed -n 's/^.*sub.*$0x\([0-9a-f]\{1,\}\),%esp/\1/p'|sort|uniq|less
>
> which shows that we have at least one place where we allocate 327,788
> bytes on the stack (!).  I hope that is not in the flow processing path!
>
> On Tue, Jul 08, 2014 at 05:36:07PM +0100, Anoob Soman wrote:
>> I have been running tests with 1MB stack size and ovs-vswitchd seem
>> to hold pretty well. I will try to do some more experiments to find
>> out the max depth of the stack, but I am afraid this will totally
>> depend on the test I am running. Any suggestion on what sort of test
>> I should be running ? More over "force-miss-model" other-config is
>> missing from 2.1.x as there is no concept of facets. Is there way
>> that I can force all packets to be processed in userspace, other
>> than me doing "ovs-dpctl del-flows" periodically.
>>
>> Thanks,
>> Anoob.
>> On 08/07/14 17:15, Ben Pfaff wrote:
>>> On Tue, Jul 08, 2014 at 05:08:43PM +0100, Anoob Soman wrote:
>>>> Since openvswitch has moved to multi-threaded model, RSS usage of
>>>> ovs-vswitchd has increased quite significantly compared to the last
>>>> release we used (ovs-1.4.x). Part of the problem is using mlockall
>>>> (with MCL_CURRENT|MCL_FUTURE) on ovs-vswitchd, which causes every
>>>> pthreads stack's and heap's virtual address to locked to RAM.
>>>> ovs-vswitch (2.1.x) running on a 8 vCPU dom0 (10 pthreads) uses
>>>> around 89M of RSS (80MB just for stack), without any VMs running on
>>>> the host. One way to reduce RSS would be to reduce the number of
>>>> "n-handler-threads" and "n-revalidator-threads", but I am not sure
>>>> about the performance impact of having these thread numbers reduced.
>>>> I am wondering if the stack size of the pthreads can be reduce
>>>> (using pthread_attr_setstack). By default pthreads max stack size is
>>>> 8MB and mlockall locks all of this 8MB into RAM. What could be
>>>> optimal stack size that can be used.
>>> I think it would be very reasonable to reduce the stack sizes, but I
>>> don't know the "correct" size off-hand.  Since you're looking at the
>>> problem already, perhaps you should consider some experiments.