[ovs-dev] [PATCH 0/6] DPDK: simplify configuration

[Daniele Di Proietto]

> On 18 Mar 2015, at 16:53, Traynor, Kevin <kevin.traynor at intel.com> wrote:
> 
>> 
>> -----Original Message-----
>> From: Daniele Di Proietto [mailto:diproiettod at vmware.com]
>> Sent: Monday, March 16, 2015 6:16 PM
>> To: Traynor, Kevin
>> Cc: dev at openvswitch.org
>> Subject: Re: [ovs-dev] [PATCH 0/6] DPDK: simplify configuration
>> 
>> 
>>> On 16 Mar 2015, at 15:18, Traynor, Kevin <kevin.traynor at intel.com> wrote:
>>> 
>>>> -----Original Message-----
>>>> From: dev [mailto:dev-bounces at openvswitch.org] On Behalf Of Daniele Di
>>>> Proietto
>>>> Sent: Thursday, March 12, 2015 6:05 PM
>>>> To: dev at openvswitch.org
>>>> Subject: [ovs-dev] [PATCH 0/6] DPDK: simplify configuration
>>>> 
>>>> This series improves OVS configuration with DPDK in three ways:
>>>> 
>>>> * netdev-dpdk is patched to work on smaller systems (without 1GB
>> hugepages)
>>>> or with smaller NICs (without lots of transmission queues).
>>>> * the 'other_config:nonpmd-cpu-mask' key is introduced: it can be used to
>>>> limit OVS non PMD threads to a particular set of cores.
>>>> * the 'other_config:n-pmd-cores' key is introduced: it allows setting the
>>>> number of PMD threads without specifing a CPU mask.
>>> 
>>> Hi, I've reviewed this patchset - few comments/questions on it…
>> 
>> Hi Kevin, thanks
>> 
>>> I haven't tested yet - but I'm wondering what is the impact to the dpdk -c
>>> parameter. Is it no longer used for OVS?
>>> 
>> 
>> Yes, that’s correct. It should have no impact on OVS. I think we
>> should also provide a default, i.e. generate the parameters passed to
>> dpdk_eal_init at some point.
>> 
>>> At present the NON_PMD_CORE_ID define overrides the db settings (which is
>>> clearly documented). Is it needed now that a db key is available? Perhaps
>>> it would make things simpler that the define is overridden when a key is
>>> specified?
>>> 
>> 
>> We check NON_PMD_CORE_ID often in the fast path.  I would prefer leaving it
>> as
>> it is right now and removing this limitation later, making sure that it has
>> no
>> impact on performance.
> 
> Hi, I've tested the core pinning. Everything worked as I expected with a 
> couple of exceptions. There was no noticeable negative effect on datapath 
> performance.
> 
> - the NON_PMD_CORE_ID is used *in addition* to the mask for nonpmd's - I think 
>  this should get a mention in the docs.

I mentioned it in vswitchd/vswitch.xml:

        <p>
          Please note that core 0 (NON_PMD_CORE_ID in the code) is always used
          for non PMD threads, even if unset in this mask.
        </p>

and in INSTALL.DPDK.md:

   `ovs-vsctl set Open_vSwitch . other_config:nonpmd-cpu-mask=<hex string>`

   The above command will limit OVS non PMD threads on the CPU set specified
   by the hex string.  Please note that core 0 is always used by non PMD
   threads, even if it is unset in this mask. For more information,
   please refer to `man ovs-vswitchd.conf.db`

I thought it was clear enough, but we can always improve it

> - I haven't done it before, so I'm not sure if it's related to these changes, 
>  but I wasn't able to get pmd's setup on the second socket. Getting this 
>  error "dpif_netdev|ERR|Cannot create pmd threads due to out of unpinned 
>  cores on numa node”.

If you’re trying to use a NIC attached to NUMA socket 0 and you’re specifying
a CPU mask without any core attached to NUMA socket 0, then the error makes sense.
If you specify a CPU mask I expect that you know what you’re doing, you know about
NUMA and stuff.
To prevent this kind of errors the series introduces the far simpler ’n-pmd-cores’
parameter: the user doesn’t need to be aware of the NUMA configuration at all.

Does this make sense?

> - There is still a few requirements regarding the -c option as it's not 
>  completely 'don't care' now. e.g. it can't be set to 0. These would be 
>  redundant if you do remove from it the cmd line - if they end up staying 
>  I can update the docs.

You’re right, it cannot be zero. I was hoping to remove it anyway.

Thanks for the testing,

Daniele