[ovs-dev] dpif-netdev: Assign ports to pmds on non-local numa node.

[O Mahony, Billy]

Hi Ilya,

Comments below.

BR, 
Billy.

> -----Original Message-----
> From: Ilya Maximets [mailto:i.maximets at samsung.com]
> Sent: Tuesday, February 28, 2017 12:49 PM
> To: O Mahony, Billy <billy.o.mahony at intel.com>; dev at openvswitch.org
> Cc: Daniele Di Proietto <diproiettod at ovn.org>
> Subject: Re: [ovs-dev] dpif-netdev: Assign ports to pmds on non-local numa
> node.
> 
> On 28.02.2017 14:43, O Mahony, Billy wrote:
> > Hi Ilya,
> >
> > Thanks for the quick response. You make some good points.
> >
> > It's important to point out that local pmds are still chosen when available.
> > The change only operates to avoid totally non-operational/ non-polled
> ports.
> >
> > This is something we came across when deploying DPDK-enabled OVS in
> > OpenStack environments (OPNFV project). Where we had remote (both
> > physically and administratively) multi-node labs already wired up and
> > would have much preferred to have sub-optimal operation that a non-
> operational OpenStack environment.
> >
> > Some further comments below.
> >
> > Best Regards,
> > Billy
> >
> >> -----Original Message-----
> >> From: Ilya Maximets [mailto:i.maximets at samsung.com]
> >> Sent: Tuesday, February 28, 2017 11:21 AM
> >> To: O Mahony, Billy <billy.o.mahony at intel.com>; dev at openvswitch.org
> >> Cc: Daniele Di Proietto <diproiettod at ovn.org>
> >> Subject: Re: [ovs-dev] dpif-netdev: Assign ports to pmds on non-local
> >> numa node.
> >>
> >> Hello.
> >>
> >> On 28.02.2017 13:12, Billy O'Mahony wrote:
> >>> From: billyom <billy.o.mahony at intel.com>
> >>>
> >>> Previously if there is no available (non-isolated) pmd on the numa
> >>> node for a port then the port is not polled at all. This can result
> >>> in a non-operational system until such time as nics are physically
> >>> repositioned.
> >>
> >> Why you can't just reconfigure your pmd-cpu-mask after NICs'
> repositioning?
> > [[BO'M]] The idea is to avoid having to repositioning NICs as they may
> > be in remote data-centers administered by other organisations. Also
> > this can be related to multi-node clusters where it won't be just one
> > NIC on one system that needs to be moved but several NICS/nodes
> affected.
> >
> >
> >> Maybe you can use pmd-rxq-affinity to assign port on another NUMA
> node?
> > [[BO'M]] The low-performance assignment will only occur if there is no
> > available PMD on the local NUMA node. Ie if possible the normal highly
> > performant assignment is made. It is only when it is a choice between
> > lower performance and total non-performance that the lesser of two evils
> is chosen.
> 
> Maybe you can include at least one core from each node in pmd-cpu-mask?

[[BO'M]] Ideally everyone using and developing  the many OpenStack deployment tools would be aware of the optimal settings but we can expect that this will not the case in many situations. Poor configurations will occur. A warning message is included in the VLOG in this case to give users a hint that performance will not be optimal but without imposing the penalty of a broken system - which is a heavy penalty if you don’t even have direct access to the pmd-cpu-mask as you are driving the ovs configuration via an OpenStack installation tool such as Fuel, RDO, etc.

> 
> >> The main concern here is that this 'remote' port will degrade
> >> performance of other ports served by chosen PMD thread significantly.
> > [[BO'M]] That is a good point that there are second-order consequences on
> other ports.
> > But certainly with a many cloud systems even if one port is
> > non-operational it means the entire node is effectively down - for
> > instance an OpenStack compute node with a non-working provider
> network
> > for it's tenant VMs is useless even though the administration and control
> networks are still working.
> >>
> >> Best regards, Ilya Maximets.