[ovs-dev] [PATCH v2] netdev-dpdk: Set pmd thread priority

Bhanuprakash Bodireddy bhanuprakash.bodireddy at intel.com
Tue Jul 5 20:05:57 UTC 2016


Set the DPDK pmd thread scheduling policy to SCHED_RR and static
priority to highest priority value of the policy. This is to deal with
pmd thread starvation case where another cpu hogging process can get
scheduled/affinitized on to the same core the pmd thread is running there
by significantly impacting the datapath performance.

Setting the realtime scheduling policy to the pmd threads is one step
towards Fastpath Service Assurance in OVS DPDK.

The realtime scheduling policy is applied only when CPU mask is passed
to 'pmd-cpu-mask'. The exception to this is 'pmd-cpu-mask=1', where the
policy and priority shall not be applied to pmd thread spawned on core0.
For example:

    * In the absence of pmd-cpu-mask or if pmd-cpu-mask=1, one pmd
      thread shall be created and affinitized to 'core 0' with default
      scheduling policy and priority applied.

    * If pmd-cpu-mask is specified with CPU mask > 1, one or more pmd
      threads shall be spawned on the corresponding core(s) in the mask
      and real time scheduling policy SCHED_RR and highest static
      priority is applied to the pmd thread(s).

To reproduce the issue use following commands:

ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=6
taskset 0x2 cat /dev/zero > /dev/null &

Also OVS control threads should not be affinitized to the pmd cores.
For example 'dpdk-lcore-mask' and 'pmd-cpu-mask' should be exclusive.

v1->v2:
* Removed #ifdef and introduced dummy function "pmd_thread_setpriority" 
  in netdev-dpdk.h
* Rebase

Signed-off-by: Bhanuprakash Bodireddy <bhanuprakash.bodireddy at intel.com>
---
 lib/dpif-netdev.c |  8 ++++++++
 lib/netdev-dpdk.c | 14 ++++++++++++++
 lib/netdev-dpdk.h |  7 +++++++
 3 files changed, 29 insertions(+)

diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
index 37c2631..6ff81d6 100644
--- a/lib/dpif-netdev.c
+++ b/lib/dpif-netdev.c
@@ -2849,6 +2849,14 @@ pmd_thread_main(void *f_)
     ovs_numa_thread_setaffinity_core(pmd->core_id);
     dpdk_set_lcore_id(pmd->core_id);
     poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list);
+
+    /* Set pmd thread's scheduling policy to SCHED_RR and priority to
+     * highest priority of SCHED_RR policy, In absence of pmd-cpu-mask (or)
+     * pmd-cpu-mask=1, default scheduling policy and priority shall
+     * apply to pmd thread */
+     if (pmd->core_id) {
+         pmd_thread_setpriority(SCHED_RR);
+     }
 reload:
     emc_cache_init(&pmd->flow_cache);
 
diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
index 02e2c58..ce1683b 100644
--- a/lib/netdev-dpdk.c
+++ b/lib/netdev-dpdk.c
@@ -3541,3 +3541,17 @@ dpdk_thread_is_pmd(void)
 {
     return rte_lcore_id() != NON_PMD_CORE_ID;
 }
+
+void
+pmd_thread_setpriority(int policy)
+{
+    struct sched_param threadparam;
+    int err;
+
+    memset(&threadparam, 0, sizeof(threadparam));
+    threadparam.sched_priority = sched_get_priority_max(policy);
+    err = pthread_setschedparam(pthread_self(), policy, &threadparam);
+    if (err) {
+        VLOG_WARN("Thread priority error %d",err);
+    }
+}
diff --git a/lib/netdev-dpdk.h b/lib/netdev-dpdk.h
index 80bb834..1890ae4 100644
--- a/lib/netdev-dpdk.h
+++ b/lib/netdev-dpdk.h
@@ -26,6 +26,7 @@ struct smap;
 void netdev_dpdk_register(void);
 void free_dpdk_buf(struct dp_packet *);
 void dpdk_set_lcore_id(unsigned cpu);
+void pmd_thread_setpriority(int policy);
 
 #else
 
@@ -51,6 +52,12 @@ dpdk_set_lcore_id(unsigned cpu OVS_UNUSED)
     /* Nothing */
 }
 
+static inline void
+pmd_thread_setpriority(int policy OVS_UNUSED)
+{
+    /* Nothing */
+}
+
 #endif /* DPDK_NETDEV */
 
 void dpdk_init(const struct smap *ovs_other_config);
-- 
2.4.11




More information about the dev mailing list