[ovs-dev] [ovs dpdk] why all the ovs threads pinned to master lcore?
Bodireddy, Bhanuprakash
bhanuprakash.bodireddy at intel.com
Tue Oct 25 08:45:21 UTC 2016
>
>sorry, I am still confused.
>when I launched vswitchd with dpdk-init=false, all the threads changed their
>pinned cores as time goes by.
>The following output is from ovs version 2.6 with dpdk-init=false ps -To
>tid,pid,psr,comm -p 5922
> TID PID PSR COMMAND
> 5922 5922 12 ovs-vswitchd
> 5934 5922 24 urcu5 ==>does not follow main the ovs-vswitchd
> 6917 5922 0 handler98
> 6918 5922 24 handler100
> 6919 5922 0 handler99
> 6920 5922 0 handler101
> 6921 5922 0 handler102
> 6922 5922 2 handler103
>6952 5922 0 revalidator133
> 6953 5922 10 revalidator134
> 6954 5922 38 revalidator135 ==>seems these threads can be pinned on any
>cores, just like CPU auto scheduled
> 6955 5922 26 revalidator136
> 6956 5922 40 revalidator137
> 6957 5922 22 revalidator138
>
>
>after somtime,
>5922 5922 7 ovs-vswitchd ==>changed from core 12 to core 7
> 5934 5922 18 urcu5
> 6917 5922 0 handler98
> 6918 5922 0 handler100
> 6919 5922 0 handler99
> 6920 5922 0 handler101
> 6921 5922 0 handler102
> 6922 5922 0 handler103
>6952 5922 3 revalidator133
> 6953 5922 35 revalidator134
> 6954 5922 27 revalidator135
> 6955 5922 21 revalidator136
> 6956 5922 9 revalidator137
> 6957 5922 33 revalidator138
>
>
>When launch vswitchd with dpdk-init as true, all the handler threads and
>revalidator threads are pinned to master lcore and can not change their
>pinned cores when time goes by.
>so here is my problem:
>1. with the same ovs code, there is different phenomenon for ovs threads
>with dpdk inited have different values?
Yes, with dpdk_init=true, the dpdk initialization will be done and there is a mechanism to determine and pin
the threads to the cores. If 'dpdk-lcore-mask' is specified the threads would be pinned to the lowest core of
the mask. This is more dpdk library behavior as DPDK wants to use the lowest core of the mask as control core
and the rest of the cores as packet forwarding cores.
If one doesn't specifiy the dpdk-lcore-mask, the thread will not be pinned explicitly to any core.
To check this run taskset on the respective threads to retrieve the affinity In this case you can see
the threads are free to float on any core between 0 and 27. For example.
$ ps -eLo tid,psr,comm | grep -e revalidator -e handler -e ovs -e pmd -e urc -e eal
99078 17 ovs-vswitchd
99137 11 handler33
99157 6 revalidator53
$taskset -cp 99078
pid 99078's current affinity list: 0-27
$ taskset -cp 99137
pid 99137's current affinity list: 0-27
$ taskset -cp 99157
pid 99157's current affinity list: 0-27
Hope this clears your confusion.
Regards,
Bhanu Prakash.
>2.with dpdk-init=true, if all the other threads share the same logical core, then
>for the performance meaning, is it not friendly?
>
>
>
>
>
>
>
>
>
>
>At 2016-10-25 01:39:13, "Kevin Traynor" <ktraynor at redhat.com> wrote:
>>On 10/24/2016 11:55 AM, ychen wrote:
>>> hi, I am a freshman to ovs DPDK, when I tried to launch ovs with dpdk
>>> inited, I found that all the ovs threads are pinned to master lcore, but I
>can't find any code for setting the affinity of the specified thread.
>>
>>On older versions of OVS you you can set the affinity via the -c
>>0x<mask> vswitchd dpdk cmd line arg. It will use the lsb only. For the
>>latest versions, you can set this through OVSDB - alternatively you can
>>not set it and by default the non-pmd threads will float on the cores
>>that vswitchd runs on.
>>
>>> Here is my configuration:
>>> lscpu
>>> Architecture: x86_64
>>> CPU op-mode(s): 32-bit, 64-bit
>>> Byte Order: Little Endian
>>> CPU(s): 48
>>> On-line CPU(s) list: 0-47
>>> Thread(s) per core: 2
>>> Core(s) per socket: 12
>>> Socket(s): 2
>>> NUMA node(s): 2
>>> Vendor ID: GenuineIntel
>>> CPU family: 6
>>> Model: 63
>>> Stepping: 2
>>> CPU MHz: 2599.988
>>> BogoMIPS: 4600.75
>>> Virtualization: VT-x
>>> L1d cache: 32K
>>> L1i cache: 32K
>>> L2 cache: 256K
>>> L3 cache: 30720K
>>> NUMA node0 CPU(s):
>0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46
>>> NUMA node1 CPU(s):
>1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47
>>>
>>>
>>> ovs-vsctl list open_vswitch
>>> _uuid : 61d9066b-178e-4672-b8b3-9dcb5587565d
>>> bridges : [cc2605fb-fcbb-4627-8834-080c43534119]
>>> cur_cfg : 68
>>> datapath_types : [netdev, system]
>>> db_version : []
>>> external_ids : {}
>>> iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient,
>geneve, gre, internal, ipsec_gre, lisp, patch, stt, system, tap, vxlan]
>>> manager_options : []
>>> next_cfg : 68
>>> other_config : {dpdk-init="true", dpdk-lcore-mask="0xf", dpdk-socket-
>mem="1024,1024", pmd-cpu-mask="f0"}
>>> ovs_version : []
>>> ssl : []
>>> statistics : {}
>>> system_type : []
>>> system_version : []
>>>
>>>
>>> ps -To tid,pid,psr,comm -p 28262
>>> TID PID PSR COMMAND
>>> 28262 28262 0 ovs-vswitchd
>>> 28263 28262 39 vfio-sync
>>> 28297 28262 0 eal-intr-thread
>>> 28298 28262 1 lcore-slave-1
>>> 28299 28262 2 lcore-slave-2
>>> 28300 28262 3 lcore-slave-3
>>> 28301 28262 0 dpdk_watchdog2
>>> 28302 28262 0 vhost_thread1
>>> 28303 28262 0 pdump-thread
>>> 28304 28262 0 ct_clean3
>>> 28305 28262 0 urcu4
>>> 28744 28262 0 handler101
>>> 28745 28262 0 handler100
>>> 28746 28262 0 handler99
>>> 28747 28262 0 handler98
>>> 28748 28262 0 handler95
>>> 28749 28262 0 handler77
>>> 28750 28262 0 handler79
>>> 28751 28262 0 handler80
>>> 28752 28262 0 handler81
>>> 28753 28262 0 handler73
>>> 28756 28262 0 handler92
>>> 28757 28262 0 handler82
>>> 28758 28262 0 handler96
>>> 28759 28262 0 handler71
>>> 28760 28262 0 handler61
>>> 28761 28262 0 handler62
>>> 28762 28262 0 handler83
>>> 28763 28262 0 handler63
>>> 28764 28262 0 handler84
>>> 28765 28262 0 handler93
>>> 28766 28262 0 handler64
>>> 28767 28262 0 handler85
>>> 28768 28262 0 handler74
>>> 28769 28262 0 handler65
>>> 28770 28262 0 handler66
>>> 28771 28262 0 handler78
>>> 28772 28262 0 handler86
>>> 28773 28262 0 handler87
>>> 28774 28262 0 handler97
>>> 28775 28262 0 handler88
>>> 28776 28262 0 handler56
>>> 28777 28262 0 handler76
>>> 28778 28262 0 handler67
>>> 28779 28262 0 handler60
>>> 28780 28262 0 handler68
>>> 28781 28262 0 revalidator75
>>> 28782 28262 0 revalidator57
>>> 28783 28262 0 revalidator89
>>> 28784 28262 0 revalidator69
>>> 28785 28262 0 revalidator54
>>> 28786 28262 0 revalidator90
>>> 28787 28262 0 revalidator55
>>> 28788 28262 0 revalidator58
>>> 28789 28262 0 revalidator59
>>> 28790 28262 0 revalidator70
>>> 28791 28262 0 revalidator94
>>> 28792 28262 0 revalidator91
>>> 28793 28262 0 revalidator72
>>> 28827 28262 4 pmd103
>>> 28829 28262 6 pmd102
>>> _______________________________________________
>>> dev mailing list
>>> dev at openvswitch.org
>>> http://openvswitch.org/mailman/listinfo/dev
>>>
>>
>_______________________________________________
>dev mailing list
>dev at openvswitch.org
>http://openvswitch.org/mailman/listinfo/dev
More information about the dev
mailing list