[ovs-dev] [ovs dpdk] why all the ovs threads pinned to master lcore?

[Kevin Traynor]

On 10/25/2016 03:15 PM, Bodireddy, Bhanuprakash wrote:
>> -----Original Message-----
>> From: Aaron Conole [mailto:aconole at redhat.com]
>> Sent: Tuesday, October 25, 2016 2:36 PM
>> To: Bodireddy, Bhanuprakash <bhanuprakash.bodireddy at intel.com>
>> Cc: Christian Ehrhardt <christian.ehrhardt at canonical.com>; ychen
>> <ychen103103 at 163.com>; dev at openvswitch.org
>> Subject: Re: [ovs-dev] [ovs dpdk] why all the ovs threads pinned to master
>> lcore?
>>
>> "Bodireddy, Bhanuprakash" <bhanuprakash.bodireddy at intel.com> writes:
>>
>>>>
>>>> ++ pidof ovs-vswitchd
>>>> + ps -To tid,pid,psr,comm -p 25932
>>>>  TID   PID PSR COMMAND
>>>> 25932 25932   1 ovs-vswitchd
>>>> 25934 25932   0 eal-intr-thread
>>>> 25935 25932   1 dpdk_watchdog1
>>>> 25936 25932   2 vhost_thread2
>>>> 25937 25932   1 pdump-thread
>>>> 25938 25932   8 urcu3
>>>> 25939 25932   2 ovs-vswitchd
>>>> 25959 25932   1 ct_clean4
>>>> 26013 25932   3 handler18
>>>> 26014 25932   9 handler17
>>>> 26015 25932   3 handler20
>>>> 26016 25932  11 handler19
>>>> 26022 25932   4 handler21
>>>> 26028 25932   9 handler22
>>>> 26033 25932  10 handler23
>>>> 26037 25932  10 handler24
>>>> 26042 25932   9 revalidator25
>>>> 26045 25932  10 revalidator26
>>>> 26046 25932   0 revalidator27
>>>> 26054 25932   2 revalidator28
>>>> 26143 25932   5 pmd29
>>>> 26144 25932   4 pmd30
>>>
>>> This is for the reason that 'dpdk-lcore-mask' hasn't been specified in
>>> the above case.
>>> If one specify dpdk-lcore-mask=<mask>, all the
>>> threads(revalidator/handler/ovs-vswitchd/urc) will automatically be
>>> pinned to the lowest core of the 'mask'.
>>>
>>> For example with 'ovs-vsctl --no-wait set Open_vSwitch .
>>> other_config:dpdk-lcore-mask=f00', I would expect the threads to float
>>> between cores 8 and 11, but all the threads get pinned to lowest core
>>> of the core-mask which is core 8.  Hope this clears the confusion.
>>
>> I'm sorry, I don't understand this behaviors.  I don't see that this is anything to
>> do with ovs, specifically.  Meaning, when that option is specified, we pass it to
>> the dpdk library directly, and take no actions as far as thread binding is
>> concerned.  I have to admit, I've not delved that deeply into the dpdk side.
> 
> Completely agree Aaron. This isn't anything to do with OVS.  This is the current behavior of DPDK and more to do with how
> it parses the coremask. For the given coremask, DPDK assumes the lowest core of the mask as 'mastercore/control core'
> and pins the control threads to this core, leaving the remaining cores for actual packet forwarding. 

I take the opposite view :) this is everything to do with OVS as it
doesn't use DPDK for multi-threading, it creates its own threads. We
strip the other bits from the mask before it is sent to EAL init as
otherwise they will create worker threads which OVS will never assign
anything to. Handler/Revalidator threads then get create and pinned to
the master thread.

But the above is *not* the default behavior and the default behavior is
much nicer IMO. If dpdk-lcore-mask is not specified the
handler/revalidator type threads will float on the cores that vswitchd
runs on - similar to OVS without DPDK.

> 
> This behavior is reproducible with DPDK sample app.
> 
> Regards,
> Bhanu Prakash. 
> 
>>
>> Do you have the output from the EAL logs?  There are logs which state what
>> dpdk thinks it should have set for an affinity.  Perhaps those shed some light?
>>
>> -Aaron
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> http://openvswitch.org/mailman/listinfo/dev
>