[ovs-dev] [PATCH] INSTALL.DPDK.md: Clarify DPDK arguments.

[Zoltan Kiss]

Hi,

On 14/12/15 16:02, Aaron Conole wrote:
> Just some more activity on this topic, after spending time going through
> the code a bit.
>
> Zoltan Kiss <zoltan.kiss at linaro.org> writes:
>> On 08/12/15 17:31, Gray, Mark D wrote:
>>>
>>>
>>>> -----Original Message-----
>>>> From: Ben Pfaff [mailto:blp at ovn.org]
>>>> Sent: Tuesday, December 8, 2015 4:50 PM
>>>> To: Gray, Mark D
>>>> Cc: Traynor, Kevin; dev at openvswitch.org
>>>> Subject: Re: [ovs-dev] [PATCH] INSTALL.DPDK.md: Clarify DPDK arguments.
>>>>
>>>> On Tue, Dec 08, 2015 at 04:41:37PM +0000, Gray, Mark D wrote:
>>>>>> -----Original Message-----
>>>>>> From: dev [mailto:dev-bounces at openvswitch.org] On Behalf Of Kevin
>>>>>> Traynor
>>>>>> Sent: Monday, December 7, 2015 5:58 PM
>>>>>> To: dev at openvswitch.org
>>>>>> Subject: [ovs-dev] [PATCH] INSTALL.DPDK.md: Clarify DPDK arguments.
>>>>>>
>>>>>> Add some information about the DPDK -c and -n parameters.
>>>>>>
>>>>>> Signed-off-by: Kevin Traynor <kevin.traynor at intel.com>
>>>>>> Reported-by: Zoltan Kiss <zoltan.kiss at linaro.org>
>>>>>> ---
>>>>>>    INSTALL.DPDK.md |   14 ++++++++++++--
>>>>>>    1 files changed, 12 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index
>>>>>> 96b686c..ee016da
>>>>>> 100644
>>>>>> --- a/INSTALL.DPDK.md
>>>>>> +++ b/INSTALL.DPDK.md
>>>>>> @@ -145,8 +145,18 @@ Using the DPDK with ovs-vswitchd:
>>>>>>
>>>>>>       DPDK configuration arguments can be passed to vswitchd via `--dpdk`
>>>>>>       argument. This needs to be first argument passed to vswitchd process.
>>>>>> -   dpdk arg -c is ignored by ovs-dpdk, but it is a required parameter
>>>>>> -   for dpdk initialization.
>>>>>> +   The DPDK configuration arguments are passed to DPDK during DPDK
>>>>>> +   initialization.
>>>>>> +
>>>>>> +   The DPDK -c coremask is a required argument. To avoid wasted
>>>> resources
>>>>>> +   only one core should be set. The standard OVS threads (e.g. main
>>>>>> +   process, handler, revalidator) will run on the core that is specified.
>>>>>
>>>>> Might be worth mentioning that then there is a corresponding potential
>>>>> decrease in performance of revalidation and flow handling.
>>>>
>>>> With the kernel datapath, OVS sets up flows and revalidates them on
>>>> multiple cores.  You're saying that with DPDK it only uses one core?
>>>> Why?
>>>
>>> In my opinion, the core mask should define the affinities of the other
>>> threads (main, process, handler, revalidator) and the pmd-cpu-mask
>>
>> That would sound good, but I'm afraid you can't really influence how
>> DPDK interprets the -c parameter. Maybe you can delegate the tasks of
>> the non-pmd threads to the lcore threads (via
>> rte_eal_[mp_]remote_launch() functions), but I'm not sure if its
>> viable.
>
> I'm not sure how good that sounds, really. From a reading of the code,
> the coremask here is used _solely_ for the EAL 'lcore' thread(s) which
> process EAL 'rpc' messages. There'd be a lot of intrusive conversion to
> integrate with DPDK in that space.

I think the current way of handling this almost OK:

- OVS creates the threads by itself, based on pmd-cpu-mask
- the DPDK code makes sure rte_lcore_id() is set in 
pmd_thread_setaffinity_cpu(). So other DPDK libraries will see these OVS 
PMD threads as perfectly fine DPDK lcore threads
- the only thing I would change is how the "-c coremask" parameter is 
handled. Currently the user passes is, but it really should have one bit 
set in there, for the master DPDK thread. On every other core in that 
list DPDK will create an lcore thread, which won't do any work at all. 
It's not a big issue, but they still waste some resources.


>
> I think it may be a good idea to rename it here from a generic
> "coremask," but as you indicate, we can't impose any additional
> semantics without getting everyone really confused.
>
>>> should define the affinities of the packet processing threads. However,
>>> I don't know if this was the intended behavior because the name is a little
>>> too generic ("core mask").  If this was the intended behavior, Kevin and I
>>> just did some tests, and it is not behaving like this.
>
> I think the behavior intention was to leave it as 'dpdk lcore only.'
> Maybe there's a better way of doing it that I don't see, though;
> otherwise you've kindof overloaded the dpdk argument with ovs argument
> as well. Sorry if that doesn't make sense.
>
> -Aaron
>