[ovs-dev] [PATCH v5] netdev-dpdk: Set pmd thread priority

Kavanagh, Mark B mark.b.kavanagh at intel.com
Wed Jul 27 12:28:58 UTC 2016


>
>Set the DPDK pmd thread scheduling policy to SCHED_RR and static
>priority to highest priority value of the policy. This is to deal with
>pmd thread starvation case where another cpu hogging process can get
>scheduled/affinitized on to the same core the pmd thread is running
>there by significantly impacting the datapath performance.
>
>Setting the realtime scheduling policy to the pmd threads is one step
>towards Fastpath Service Assurance in OVS DPDK.
>
>The realtime scheduling policy is applied only when CPU mask is passed
>to 'pmd-cpu-mask'. For example:
>
>    * In the absence of pmd-cpu-mask, one pmd thread shall be created
>      and default scheduling policy and priority gets applied.
>
>    * If pmd-cpu-mask is specified, one or more pmd threads shall be
>      spawned on the corresponding core(s) in the mask and real time
>      scheduling policy SCHED_RR and highest priority of the policy is
>      applied to the pmd thread(s).
>
>To reproduce the pmd thread starvation case:
>
>ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=6
>taskset 0x2 cat /dev/zero > /dev/null &
>
>With this commit, it is recommended that the OVS control thread and pmd
>thread shouldn't be pinned to same core ('dpdk-lcore-mask','pmd-cpu-mask'
>should be non-overlapping). Also other processes with same affinity as
>PMD thread will be unresponsive.
>
>Signed-off-by: Bhanuprakash Bodireddy <bhanuprakash.bodireddy at intel.com>

LGTM - Acked-by: mark.b.kavanagh at intel.com

>---
>v4->v5:
>* Reword Note section in DPDK-ADVANCED.md
>
>v3->v4:
>* Document update
>* Use ovs_strerror for reporting errors in lib-numa.c
>
>v2->v3:
>* Move set_priority() function to lib/ovs-numa.c
>* Apply realtime scheduling policy and priority to pmd thread only if
>  pmd-cpu-mask is passed.
>* Update INSTALL.DPDK-ADVANCED.
>
>v1->v2:
>* Removed #ifdef and introduced dummy function "pmd_thread_setpriority"
>  in netdev-dpdk.h
>* Rebase
>
> INSTALL.DPDK-ADVANCED.md | 17 +++++++++++++----
> lib/dpif-netdev.c        |  9 +++++++++
> lib/ovs-numa.c           | 18 ++++++++++++++++++
> lib/ovs-numa.h           |  1 +
> 4 files changed, 41 insertions(+), 4 deletions(-)
>
>diff --git a/INSTALL.DPDK-ADVANCED.md b/INSTALL.DPDK-ADVANCED.md
>index 9ae536d..d76cb4e 100644
>--- a/INSTALL.DPDK-ADVANCED.md
>+++ b/INSTALL.DPDK-ADVANCED.md
>@@ -205,8 +205,10 @@ needs to be affinitized accordingly.
>     pmd thread is CPU bound, and needs to be affinitized to isolated
>     cores for optimum performance.
>
>-    By setting a bit in the mask, a pmd thread is created and pinned
>-    to the corresponding CPU core. e.g. to run a pmd thread on core 2
>+    By setting a bit in the mask, a pmd thread is created, pinned
>+    to the corresponding CPU core and the scheduling policy SCHED_RR
>+    along with maximum priority of the policy applied to the pmd thread.
>+    e.g. to pin a pmd thread on core 2
>
>     `ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=4`
>
>@@ -234,8 +236,10 @@ needs to be affinitized accordingly.
>   responsible for different ports/rxq's. Assignment of ports/rxq's to
>   pmd threads is done automatically.
>
>-  A set bit in the mask means a pmd thread is created and pinned
>-  to the corresponding CPU core. e.g. to run pmd threads on core 1 and 2
>+  A set bit in the mask means a pmd thread is created, pinned to the
>+  corresponding CPU core and the scheduling policy SCHED_RR with highest
>+  priority of the scheduling policy applied to pmd thread.
>+  e.g. to run pmd threads on core 1 and 2
>
>   `ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=6`
>
>@@ -246,6 +250,11 @@ needs to be affinitized accordingly.
>
>   NIC port0 <-> OVS <-> VM <-> OVS <-> NIC port 1
>
>+  Note: It is recommended that the OVS control thread and pmd thread
>+  shouldn't be pinned to same core i.e 'dpdk-lcore-mask' and 'pmd-cpu-mask'
>+  cpu mask settings should be non-overlapping. Also other processes with
>+  same affinity as PMD thread will be unresponsive.
>+
> ### 4.3 DPDK physical port Rx Queues
>
>   `ovs-vsctl set Interface <DPDK interface> options:n_rxq=<integer>`
>diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
>index f05ca4e..b85600b 100644
>--- a/lib/dpif-netdev.c
>+++ b/lib/dpif-netdev.c
>@@ -2841,6 +2841,15 @@ pmd_thread_main(void *f_)
>     ovs_numa_thread_setaffinity_core(pmd->core_id);
>     dpdk_set_lcore_id(pmd->core_id);
>     poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list);
>+
>+    /* When cpu affinity mask explicitly set using pmd-cpu-mask, pmd thread's
>+     * scheduling policy is set to SCHED_RR and the priority to highest priority
>+     * of SCHED_RR policy.  In the absence of pmd-cpu-mask, default scheduling
>+     * policy and priority shall apply to pmd thread.
>+     */
>+    if (pmd->dp->pmd_cmask) {
>+        ovs_numa_thread_setpriority(SCHED_RR);
>+    }
> reload:
>     emc_cache_init(&pmd->flow_cache);
>
>diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c
>index c8173e0..428f274 100644
>--- a/lib/ovs-numa.c
>+++ b/lib/ovs-numa.c
>@@ -613,3 +613,21 @@ int ovs_numa_thread_setaffinity_core(unsigned core_id OVS_UNUSED)
>     return EOPNOTSUPP;
> #endif /* __linux__ */
> }
>+
>+void
>+ovs_numa_thread_setpriority(int policy)
>+{
>+    if (dummy_numa) {
>+        return;
>+    }
>+
>+    struct sched_param threadparam;
>+    int err;
>+
>+    memset(&threadparam, 0, sizeof(threadparam));
>+    threadparam.sched_priority = sched_get_priority_max(policy);
>+    err = pthread_setschedparam(pthread_self(), policy, &threadparam);
>+    if (err) {
>+        VLOG_ERR("Thread priority error %s",ovs_strerror(err));
>+    }
>+}
>diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h
>index be836b2..94f0884 100644
>--- a/lib/ovs-numa.h
>+++ b/lib/ovs-numa.h
>@@ -56,6 +56,7 @@ void ovs_numa_unpin_core(unsigned core_id);
> struct ovs_numa_dump *ovs_numa_dump_cores_on_numa(int numa_id);
> void ovs_numa_dump_destroy(struct ovs_numa_dump *);
> int ovs_numa_thread_setaffinity_core(unsigned core_id);
>+void ovs_numa_thread_setpriority(int policy);
>
> #define FOR_EACH_CORE_ON_NUMA(ITER, DUMP)                    \
>     LIST_FOR_EACH((ITER), list_node, &(DUMP)->dump)
>--
>2.4.11




More information about the dev mailing list