[ovs-dev] [PATCH v8] netdev-dpdk: Increase pmd thread priority
Bodireddy, Bhanuprakash
bhanuprakash.bodireddy at intel.com
Fri May 26 14:30:56 UTC 2017
Hi Billy,
>Hi Bhanu,
>
>This patch no longer applies cleanly.
Thanks for looking in to this patch. It's pretty old patch and should be rebased.
>
>$git apply ...
>error: Documentation/intro/install/dpdk-advanced.rst: No such file or
>directory
>error: patch failed: lib/ovs-numa.h:56
>error: lib/ovs-numa.h: patch does not apply
>
>Also some more information on the rationale behind such a change would be
>useful. E.g. is it helpful in the case where the PMDs have not been given
>isolated cores?
As you would have already noticed, this patch was at v8.
Initially I started out with applying real time scheduling policy and priority to PMD threads. It was noticed that real time priorities(the original implementation) could potentially cause problems in few corner cases. Flavio and Daniele suggested to bump up the PMD thread priority instead of applying real time scheduling policy.
This is what was implemented in the last sent v8. With datapath threads having higher priority the overall switching performance wouldn't suffer and may improve in out-of-box deployments as mentioned by Flavio in other mail.
I will rebase and send the patch.
- Bhanuprakash.
>
>> -----Original Message-----
>> From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
>> bounces at openvswitch.org] On Behalf Of Aaron Conole
>> Sent: Tuesday, January 3, 2017 8:08 PM
>> To: Bodireddy, Bhanuprakash <bhanuprakash.bodireddy at intel.com>
>> Cc: dev at openvswitch.org
>> Subject: Re: [ovs-dev] [PATCH v8] netdev-dpdk: Increase pmd thread
>> priority
>>
>> Bhanuprakash Bodireddy <bhanuprakash.bodireddy at intel.com> writes:
>>
>> > Increase the DPDK pmd thread scheduling priority by lowering the
>> > nice value. This will advise the kernel scheduler to prioritize pmd
>> > thread over other processes.
>> >
>> > Signed-off-by: Bhanuprakash Bodireddy
>> > <bhanuprakash.bodireddy at intel.com>
>> > ---
>>
>> Sorry for jumping into this so late. Is there a measured benefit to this
>patch?
>> Do you have a test case to reproduce the effect you're seeing? Might
>> it be better to write up documentation for the user describing
>> chrt/nice/renice utilities?
>>
>> > v7->v8:
>> > * Rebase
>> > * Update the documentation file
>> > @Documentation/intro/install/dpdk-advanced.rst
>> >
>> > v6->v7:
>> > * Remove realtime scheduling policy logic.
>> > * Increase pmd thread scheduling priority by lowering nice value to -20.
>> > * Update doc accordingly.
>> >
>> > v5->v6:
>> > * Prohibit spawning pmd thread on the lowest core in dpdk-lcore-mask if
>> > lcore-mask and pmd-mask affinity are identical.
>> > * Updated Note section in INSTALL.DPDK-ADVANCED doc.
>> > * Tested below cases to verify system stability with pmd priority
>> > patch
>> >
>> > v4->v5:
>> > * Reword Note section in DPDK-ADVANCED.md
>> >
>> > v3->v4:
>> > * Document update
>> > * Use ovs_strerror for reporting errors in lib-numa.c
>> >
>> > v2->v3:
>> > * Move set_priority() function to lib/ovs-numa.c
>> > * Apply realtime scheduling policy and priority to pmd thread only if
>> > pmd-cpu-mask is passed.
>> > * Update INSTALL.DPDK-ADVANCED.
>> >
>> > v1->v2:
>> > * Removed #ifdef and introduced dummy function
>> "pmd_thread_setpriority"
>> > in netdev-dpdk.h
>> > * Rebase
>> >
>> > Documentation/intro/install/dpdk-advanced.rst | 8 +++++++-
>> > lib/dpif-netdev.c | 4 ++++
>> > lib/ovs-numa.c | 19 +++++++++++++++++++
>> > lib/ovs-numa.h | 1 +
>> > 4 files changed, 31 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/Documentation/intro/install/dpdk-advanced.rst
>> > b/Documentation/intro/install/dpdk-advanced.rst
>> > index 44d1cd7..67815ac 100644
>> > --- a/Documentation/intro/install/dpdk-advanced.rst
>> > +++ b/Documentation/intro/install/dpdk-advanced.rst
>> > @@ -238,7 +238,8 @@ affinitized accordingly.
>> > to be affinitized to isolated cores for optimum performance.
>> >
>> > By setting a bit in the mask, a pmd thread is created and pinned
>> > to the
>> > - corresponding CPU core. e.g. to run a pmd thread on core 2::
>> > + corresponding CPU core and the nice value set to '-20'.
>> > + e.g. to run a pmd thread on core 2::
>> >
>> > $ ovs-vsctl set Open_vSwitch . other_config:pmd-cpu-mask=0x4
>> >
>> > @@ -278,6 +279,11 @@ improvements as there will be more total CPU
>> occupancy available::
>> >
>> > NIC port0 <-> OVS <-> VM <-> OVS <-> NIC port 1
>> >
>> > + .. note::
>> > + It is recommended that the OVS control thread and pmd thread
>> shouldn't be
>> > + pinned to the same core i.e 'dpdk-lcore-mask' and
>> > + 'pmd-cpu-mask' cpu
>> mask
>> > + settings should be non-overlapping.
>> > +
>> > DPDK Physical Port Rx Queues
>> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> >
>> > diff --git a/lib/dpif-netdev.c b/lib/dpif-netdev.c index
>> > 0b73056..f58a855 100644
>> > --- a/lib/dpif-netdev.c
>> > +++ b/lib/dpif-netdev.c
>> > @@ -3120,6 +3120,10 @@ pmd_thread_main(void *f_)
>> > ovs_numa_thread_setaffinity_core(pmd->core_id);
>> > dpdk_set_lcore_id(pmd->core_id);
>> > poll_cnt = pmd_load_queues_and_ports(pmd, &poll_list);
>> > +
>> > + /* Set pmd thread's nice value to -20 */ #define MIN_NICE -20
>> > + ovs_numa_thread_setpriority(MIN_NICE);
>> > reload:
>> > emc_cache_init(&pmd->flow_cache);
>> >
>> > diff --git a/lib/ovs-numa.c b/lib/ovs-numa.c index e1e7068..d52c517
>> > 100644
>> > --- a/lib/ovs-numa.c
>> > +++ b/lib/ovs-numa.c
>> > @@ -23,6 +23,7 @@
>> > #include <dirent.h>
>> > #include <stddef.h>
>> > #include <string.h>
>> > +#include <sys/resource.h>
>> > #include <sys/types.h>
>> > #include <unistd.h>
>> > #endif /* __linux__ */
>> > @@ -620,3 +621,21 @@ int ovs_numa_thread_setaffinity_core(unsigned
>> core_id OVS_UNUSED)
>> > return EOPNOTSUPP;
>> > #endif /* __linux__ */
>> > }
>> > +
>> > +void
>> > +ovs_numa_thread_setpriority(int nice OVS_UNUSED) {
>> > + if (dummy_numa) {
>> > + return;
>> > + }
>> > +
>> > +#ifndef _WIN32
>> > + int err;
>> > + err = setpriority(PRIO_PROCESS, 0, nice);
>> > + if (err) {
>> > + VLOG_ERR("Thread priority error %s",ovs_strerror(err));
>> > + }
>> > +#else
>> > + return EOPNOTSUPP;
>> > +#endif
>> > +}
>> > diff --git a/lib/ovs-numa.h b/lib/ovs-numa.h index be836b2..3ce13c7
>> > 100644
>> > --- a/lib/ovs-numa.h
>> > +++ b/lib/ovs-numa.h
>> > @@ -56,6 +56,7 @@ void ovs_numa_unpin_core(unsigned core_id);
>> > struct ovs_numa_dump *ovs_numa_dump_cores_on_numa(int
>numa_id);
>> > void ovs_numa_dump_destroy(struct ovs_numa_dump *); int
>> > ovs_numa_thread_setaffinity_core(unsigned core_id);
>> > +void ovs_numa_thread_setpriority(int nice);
>> >
>> > #define FOR_EACH_CORE_ON_NUMA(ITER, DUMP) \
>> > LIST_FOR_EACH((ITER), list_node, &(DUMP)->dump)
>> _______________________________________________
>> dev mailing list
>> dev at openvswitch.org
>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
More information about the dev
mailing list