[ovs-dev] [ovs dpdk] why all the ovs threads pinned to master lcore?

Bodireddy, Bhanuprakash bhanuprakash.bodireddy at intel.com
Tue Oct 25 08:45:21 UTC 2016


>
>sorry, I am still confused.
>when I launched vswitchd with dpdk-init=false, all the threads changed their
>pinned cores as time goes by.
>The following output is from ovs version 2.6 with dpdk-init=false ps -To
>tid,pid,psr,comm -p 5922
>   TID    PID PSR COMMAND
>  5922   5922  12 ovs-vswitchd
>  5934   5922  24 urcu5     ==>does not follow main the ovs-vswitchd
>  6917   5922   0 handler98
>  6918   5922  24 handler100
>  6919   5922   0 handler99
>  6920   5922   0 handler101
>  6921   5922   0 handler102
>  6922   5922   2 handler103
>6952   5922   0 revalidator133
>  6953   5922  10 revalidator134
>  6954   5922  38 revalidator135  ==>seems these threads can be pinned on any
>cores, just like CPU auto scheduled
>  6955   5922  26 revalidator136
>  6956   5922  40 revalidator137
>  6957   5922  22 revalidator138
>
>
>after somtime,
>5922   5922   7 ovs-vswitchd   ==>changed from core 12 to core 7
>  5934   5922  18 urcu5
>  6917   5922   0 handler98
>  6918   5922   0 handler100
>  6919   5922   0 handler99
>  6920   5922   0 handler101
>  6921   5922   0 handler102
>  6922   5922   0 handler103
>6952   5922   3 revalidator133
>  6953   5922  35 revalidator134
>  6954   5922  27 revalidator135
>  6955   5922  21 revalidator136
>  6956   5922   9 revalidator137
>  6957   5922  33 revalidator138
>
>
>When launch vswitchd with dpdk-init as true, all the handler threads and
>revalidator threads are pinned to master lcore and can not change their
>pinned cores when time goes by.
>so here is my problem:
>1. with the same ovs code, there is different phenomenon for ovs threads
>with dpdk inited have different values?

Yes, with dpdk_init=true, the dpdk initialization will be done and there is a mechanism to determine and pin
the threads to the cores. If 'dpdk-lcore-mask' is specified the threads would be pinned to the lowest core of 
the mask. This is more dpdk library behavior as DPDK wants to use the lowest core of the mask as control core
and the rest of the cores as packet forwarding cores. 

If one doesn't specifiy the dpdk-lcore-mask, the thread will not be pinned explicitly to any core. 
To check this run taskset on the respective threads to retrieve the affinity In this case you can see
the threads are free to float on any core between 0 and 27. For example.

$ ps -eLo tid,psr,comm | grep -e revalidator -e handler -e ovs -e pmd -e urc -e eal   
   99078  17 ovs-vswitchd
   99137  11 handler33
   99157   6 revalidator53

$taskset -cp 99078
pid 99078's current affinity list: 0-27

$ taskset -cp 99137
pid 99137's current affinity list: 0-27

$ taskset -cp 99157
pid 99157's current affinity list: 0-27

Hope this clears your confusion.

Regards,
Bhanu Prakash.

>2.with dpdk-init=true, if all the other threads share the same logical core, then
>for the performance meaning, is it not friendly?
>
>
>
>
>
>
>
>
>
>
>At 2016-10-25 01:39:13, "Kevin Traynor" <ktraynor at redhat.com> wrote:
>>On 10/24/2016 11:55 AM, ychen wrote:
>>> hi, I am a freshman to ovs DPDK, when I tried to launch ovs with dpdk
>>> inited, I found that all the ovs threads are pinned to master lcore, but I
>can't find any code for setting the affinity of the specified thread.
>>
>>On older versions of OVS you you can set the affinity via the -c
>>0x<mask> vswitchd dpdk cmd line arg. It will use the lsb only. For the
>>latest versions, you can set this through OVSDB - alternatively you can
>>not set it and by default the non-pmd threads will float on the cores
>>that vswitchd runs on.
>>
>>> Here is my configuration:
>>> lscpu
>>> Architecture:          x86_64
>>> CPU op-mode(s):        32-bit, 64-bit
>>> Byte Order:            Little Endian
>>> CPU(s):                48
>>> On-line CPU(s) list:   0-47
>>> Thread(s) per core:    2
>>> Core(s) per socket:    12
>>> Socket(s):             2
>>> NUMA node(s):          2
>>> Vendor ID:             GenuineIntel
>>> CPU family:            6
>>> Model:                 63
>>> Stepping:              2
>>> CPU MHz:               2599.988
>>> BogoMIPS:              4600.75
>>> Virtualization:        VT-x
>>> L1d cache:             32K
>>> L1i cache:             32K
>>> L2 cache:              256K
>>> L3 cache:              30720K
>>> NUMA node0 CPU(s):
>0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
>>> NUMA node1 CPU(s):
>1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
>>>
>>>
>>> ovs-vsctl  list open_vswitch
>>> _uuid               : 61d9066b-178e-4672-b8b3-9dcb5587565d
>>> bridges             : [cc2605fb-fcbb-4627-8834-080c43534119]
>>> cur_cfg             : 68
>>> datapath_types      : [netdev, system]
>>> db_version          : []
>>> external_ids        : {}
>>> iface_types         : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient,
>geneve, gre, internal, ipsec_gre, lisp, patch, stt, system, tap, vxlan]
>>> manager_options     : []
>>> next_cfg            : 68
>>> other_config        : {dpdk-init="true", dpdk-lcore-mask="0xf", dpdk-socket-
>mem="1024,1024", pmd-cpu-mask="f0"}
>>> ovs_version         : []
>>> ssl                 : []
>>> statistics          : {}
>>> system_type         : []
>>> system_version      : []
>>>
>>>
>>>  ps -To tid,pid,psr,comm -p 28262
>>>    TID    PID PSR COMMAND
>>>  28262  28262   0 ovs-vswitchd
>>>  28263  28262  39 vfio-sync
>>>  28297  28262   0 eal-intr-thread
>>>  28298  28262   1 lcore-slave-1
>>>  28299  28262   2 lcore-slave-2
>>>  28300  28262   3 lcore-slave-3
>>>  28301  28262   0 dpdk_watchdog2
>>>  28302  28262   0 vhost_thread1
>>>  28303  28262   0 pdump-thread
>>>  28304  28262   0 ct_clean3
>>>  28305  28262   0 urcu4
>>>  28744  28262   0 handler101
>>>  28745  28262   0 handler100
>>>  28746  28262   0 handler99
>>>  28747  28262   0 handler98
>>>  28748  28262   0 handler95
>>>  28749  28262   0 handler77
>>>  28750  28262   0 handler79
>>>  28751  28262   0 handler80
>>>  28752  28262   0 handler81
>>>  28753  28262   0 handler73
>>>  28756  28262   0 handler92
>>>  28757  28262   0 handler82
>>>  28758  28262   0 handler96
>>>  28759  28262   0 handler71
>>>  28760  28262   0 handler61
>>>  28761  28262   0 handler62
>>>  28762  28262   0 handler83
>>>  28763  28262   0 handler63
>>>  28764  28262   0 handler84
>>>  28765  28262   0 handler93
>>>  28766  28262   0 handler64
>>>  28767  28262   0 handler85
>>>  28768  28262   0 handler74
>>>  28769  28262   0 handler65
>>>  28770  28262   0 handler66
>>>  28771  28262   0 handler78
>>>  28772  28262   0 handler86
>>>  28773  28262   0 handler87
>>>  28774  28262   0 handler97
>>>  28775  28262   0 handler88
>>>  28776  28262   0 handler56
>>>  28777  28262   0 handler76
>>>  28778  28262   0 handler67
>>>  28779  28262   0 handler60
>>>  28780  28262   0 handler68
>>>  28781  28262   0 revalidator75
>>>  28782  28262   0 revalidator57
>>>  28783  28262   0 revalidator89
>>>  28784  28262   0 revalidator69
>>>  28785  28262   0 revalidator54
>>>  28786  28262   0 revalidator90
>>>  28787  28262   0 revalidator55
>>>  28788  28262   0 revalidator58
>>>  28789  28262   0 revalidator59
>>>  28790  28262   0 revalidator70
>>>  28791  28262   0 revalidator94
>>>  28792  28262   0 revalidator91
>>>  28793  28262   0 revalidator72
>>>  28827  28262   4 pmd103
>>>  28829  28262   6 pmd102
>>> _______________________________________________
>>> dev mailing list
>>> dev at openvswitch.org
>>> http://openvswitch.org/mailman/listinfo/dev
>>>
>>
>_______________________________________________
>dev mailing list
>dev at openvswitch.org
>http://openvswitch.org/mailman/listinfo/dev


More information about the dev mailing list