[ovs-dev] [PATCH v10] netdev-dpdk: Increase pmd thread priority.

Bodireddy, Bhanuprakash bhanuprakash.bodireddy at intel.com
Mon Jun 26 12:56:25 UTC 2017


>With this change and CFS in effect, it effectively means that the dpdk control
>threads need to be on different cores than the PMD threads or the response
>latency may be too long for their control work ?
>Have we tested having the control threads on the same cpu with -20 nice for
>the pmd thread ?

Yes, I did some testing and had a reason to add the comment that recommends dpdk-lcore-mask and pmd-cpu-mask should be non-overlapping.
The testing was done with a simple script that adds and deletes 750 vHost User ports(script copied below). The time statistics are captured in this case.

   dpdk-lcore-mask |  PMD thread   | PMD NICE |  Time statistics
      unspecified                Core 3                  -20                real    1m5.610s   / user    0m0.706s/ sys     0m0.023s          [With patch]
      Core 3                            Core 3                  -20               real    2m14.089s / user    0m0.717s/ sys     0m0.017s         [with patch]
      unspecified                Core 3                     0                 real    1m5.209s   /user    0m0.711s/sys     0m0.020s            [Master]
      Core 3                            Core 3                    0                real     1m7.209s   /user    0m0.711s/sys     0m0.020s            [Master]

In all cases, if the dpdk-lcore-mask is 'unspecified' the main thread floats between the available cores(0-27 in my case).

With this patch(PMD nice value is at -20), and with main & pmd thread pinned to core 3, the port addition and deletion took twice the time. However most important thing to notice is  with active traffic and with port addition/deletion in progress, throughput drops instantly *without* the patch. In this case the vswitchd  thread consumes 7% of the CPU time at one stage there by impacting the forwarding performance.

With the patch the throughput is still affected but happens gradually. In this case the vswitchd thread was consuming not more than 2% of the CPU time and so port addition/deletion took longer time.       

>
>I see the comment is added below
>+    It is recommended that the OVS control thread and pmd thread shouldn't
>be
>+    pinned to the same core i.e 'dpdk-lcore-mask' and 'pmd-cpu-mask' cpu
>mask
>+    settings should be non-overlapping.
>
>
>I understand that other heavy threads would be a problem for PMD threads
>and we want to effectively encourage these to be on different cores in the
>situation where we are using a pmd-cpu-mask.
>However, here we are almost shutting down other threads by default on the
>same core as PMDs threads using -20 nice, even those with little cpu load but
>just needing a reasonable latency.

I had the logic of completely shutting down other threads in the early versions of this patch by assigning real time priority to the PMD thread. But that seemed too dangerous and changing nice value is safer bet. I agree that latency can go up for non-pmd threads with this patch but it’s the same problem as there are other kernel threads that runs at -20 nice value and some with 'rt' priority. 

>
>Will this aggravate the argument from some quarters that using dpdk requires
>too much cpu reservation ?
Atleast for PMD threads that are heart of packet processing in OvS-DPDK. 


More information on commands:

script to test the port addition and deletion.

$cat port_test.sh
   cmds=; for i in {1..750}; do cmds+=" -- add-port br0 dpdkvhostuser$i -- set Interface dpdkvhostuser$i type=dpdkvhostuser"; done
   ovs-vsctl $cmds

   sleep 1;

   cmds=; for i in {1..750}; do cmds+=" -- del-port br0 dpdkvhostuser$i"; done
   ovs-vsctl $cmds

$ time ./port_test.sh

dpdk-lcore-mask and pmd-cpu-mask explicitly set to CORE 3.
-------------------------------------------------------------------------------------------
$ ovs-vsctl set Open_vSwitch . other_config:dpdk-lcore-mask=8
$ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=8
$ ps -eLo tid,psr,comm | grep -e revalidator -e handler -e ovs -e pmd -e urc -e eal
   110881  20 ovsdb-server
   110892   3 ovs-vswitchd
   110976   3 pmd61
   110898   3 eal-intr-thread
   110903   3 urcu3
   110947   3 handler60

Dpdk-lcore-mask unspecified, pmd-cpu-mask explicitly set to CORE 3.
---------------------------------------------------------------------------------------------
$ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=8
$  ps -eLo tid,psr,comm | grep -e revalidator -e handler -e ovs -e pmd -e urc -e eal
    111474  14 ovsdb-server
    111483   6 ovs-vswitchd
    111566   3 pmd61
    111564  10 revalidator60
    111489   0 eal-intr-thread
    111493   8 urcu3

Regards,
Bhanuprakash.


More information about the dev mailing list