[ovs-dev] [PATCH v8] netdev-dpdk: Increase pmd thread priority

[Aaron Conole]

Bhanuprakash Bodireddy <bhanuprakash.bodireddy at intel.com> writes:

> Increase the DPDK pmd thread scheduling priority by lowering the nice
> value. This will advise the kernel scheduler to prioritize pmd thread
> over other processes.
>
> Signed-off-by: Bhanuprakash Bodireddy <bhanuprakash.bodireddy at intel.com>
> ---

Sorry for jumping into this so late.  Is there a measured benefit to
this patch?  Do you have a test case to reproduce the effect you're
seeing?  Might it be better to write up documentation for the user
describing chrt/nice/renice utilities?

> v7->v8:
> * Rebase
> * Update the documentation file @Documentation/intro/install/dpdk-advanced.rst
>
> v6->v7:
> * Remove realtime scheduling policy logic.
> * Increase pmd thread scheduling priority by lowering nice value to -20.
> * Update doc accordingly.
>
> v5->v6:
> * Prohibit spawning pmd thread on the lowest core in dpdk-lcore-mask if
>   lcore-mask and pmd-mask affinity are identical.
> * Updated Note section in INSTALL.DPDK-ADVANCED doc.
> * Tested below cases to verify system stability with pmd priority patch
>
> v4->v5:
> * Reword Note section in DPDK-ADVANCED.md
>
> v3->v4:
> * Document update
> * Use ovs_strerror for reporting errors in lib-numa.c
>
> v2->v3:
> * Move set_priority() function to lib/ovs-numa.c
> * Apply realtime scheduling policy and priority to pmd thread only if
>   pmd-cpu-mask is passed.
> * Update INSTALL.DPDK-ADVANCED.
>
> v1->v2:
> * Removed #ifdef and introduced dummy function "pmd_thread_setpriority"
>   in netdev-dpdk.h
> * Rebase
>
>  Documentation/intro/install/dpdk-advanced.rst |  8 +++++++-
>  lib/dpif-netdev.c                             |  4 ++++
>  lib/ovs-numa.c                                | 19 +++++++++++++++++++
>  lib/ovs-numa.h                                |  1 +
>  4 files changed, 31 insertions(+), 1 deletion(-)
>
> diff --git a/Documentation/intro/install/dpdk-advanced.rst b/Documentation/intro/install/dpdk-advanced.rst
> index 44d1cd7..67815ac 100644
> --- a/Documentation/intro/install/dpdk-advanced.rst
> +++ b/Documentation/intro/install/dpdk-advanced.rst
> @@ -238,7 +238,8 @@ affinitized accordingly.
>    to be affinitized to isolated cores for optimum performance.
>  
>    By setting a bit in the mask, a pmd thread is created and pinned to the
> -  corresponding CPU core. e.g. to run a pmd thread on core 2::
> +  corresponding CPU core and the nice value set to '-20'.
> +  e.g. to run a pmd thread on core 2::
>  
>        $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x4
>  
> @@ -278,6 +279,11 @@ improvements as there will be more total CPU occupancy available::
>  
>      NIC port0 <-> OVS <-> VM <-> OVS <-> NIC port 1
>  
> +  .. note::
> +    It is recommended that the OVS control thread and pmd thread shouldn't be
> +    pinned to the same core i.e 'dpdk-lcore-mask' and 'pmd-cpu-mask' cpu mask
> +    settings should be non-overlapping.
> +
>  DPDK Physical Port Rx Queues
>  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>  
> diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c
> index 0b73056..f58a855 100644
> --- a/lib/dpif-netdev.c
> +++ b/lib/dpif-netdev.c
> @@ -3120,6 +3120,10 @@ pmd_thread_main(void *f_)
>      ovs_numa_thread_setaffinity_core(pmd->core_id);
>      dpdk_set_lcore_id(pmd->core_id);
>      poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list);
> +
> +    /* Set pmd thread's nice value to -20 */
> +#define MIN_NICE -20
> +    ovs_numa_thread_setpriority(MIN_NICE);
>  reload:
>      emc_cache_init(&pmd->flow_cache);
>  
> diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c
> index e1e7068..d52c517 100644
> --- a/lib/ovs-numa.c
> +++ b/lib/ovs-numa.c
> @@ -23,6 +23,7 @@
>  #include <dirent.h>
>  #include <stddef.h>
>  #include <string.h>
> +#include <sys/resource.h>
>  #include <sys/types.h>
>  #include <unistd.h>
>  #endif /* __linux__ */
> @@ -620,3 +621,21 @@ int ovs_numa_thread_setaffinity_core(unsigned core_id OVS_UNUSED)
>      return EOPNOTSUPP;
>  #endif /* __linux__ */
>  }
> +
> +void
> +ovs_numa_thread_setpriority(int nice OVS_UNUSED)
> +{
> +    if (dummy_numa) {
> +        return;
> +    }
> +
> +#ifndef _WIN32
> +    int err;
> +    err = setpriority(PRIO_PROCESS, 0, nice);
> +    if (err) {
> +        VLOG_ERR("Thread priority error %s",ovs_strerror(err));
> +    }
> +#else
> +    return EOPNOTSUPP;
> +#endif
> +}
> diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h
> index be836b2..3ce13c7 100644
> --- a/lib/ovs-numa.h
> +++ b/lib/ovs-numa.h
> @@ -56,6 +56,7 @@ void ovs_numa_unpin_core(unsigned core_id);
>  struct ovs_numa_dump *ovs_numa_dump_cores_on_numa(int numa_id);
>  void ovs_numa_dump_destroy(struct ovs_numa_dump *);
>  int ovs_numa_thread_setaffinity_core(unsigned core_id);
> +void ovs_numa_thread_setpriority(int nice);
>  
>  #define FOR_EACH_CORE_ON_NUMA(ITER, DUMP)                    \
>      LIST_FOR_EACH((ITER), list_node, &(DUMP)->dump)