[ovs-dev] [PATCH 2/2] Adding configuration option to whitelist DPDK physical ports.

O Mahony, Billy billy.o.mahony at intel.com
Fri Dec 8 10:45:33 UTC 2017


I can confirm that using other_config:dpdk-extra  is indeed already effective to change the hugepage file prefix (admittedly without inserting a ref to the pid) and specify a pci whitelist. 

Regards,
Billy. 

> -----Original Message-----
> From: Mooney, Sean K
> Sent: Thursday, December 7, 2017 5:53 PM
> To: Chandran, Sugesh <sugesh.chandran at intel.com>; O Mahony, Billy
> <billy.o.mahony at intel.com>; dev at openvswitch.org; blp at ovn.org
> Cc: Mooney, Sean K <sean.k.mooney at intel.com>
> Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist
> DPDK physical ports.
> 
> 
> 
> > -----Original Message-----
> > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
> > bounces at openvswitch.org] On Behalf Of Chandran, Sugesh
> > Sent: Thursday, December 7, 2017 5:07 PM
> > To: O Mahony, Billy <billy.o.mahony at intel.com>; dev at openvswitch.org;
> > blp at ovn.org
> > Subject: Re: [ovs-dev] [PATCH 2/2] Adding configuration option to
> > whitelist DPDK physical ports.
> >
> >
> >
> > Regards
> > _Sugesh
> >
> > > -----Original Message-----
> > > From: O Mahony, Billy
> > > Sent: Thursday, December 7, 2017 11:47 AM
> > > To: Chandran, Sugesh <sugesh.chandran at intel.com>;
> > dev at openvswitch.org;
> > > blp at ovn.org
> > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to
> > > whitelist DPDK physical ports.
> > >
> > > Hi Sugesh,
> > >
> > > > -----Original Message-----
> > > > From: Chandran, Sugesh
> > > > Sent: Wednesday, December 6, 2017 6:23 PM
> > > > To: O Mahony, Billy <billy.o.mahony at intel.com>;
> > dev at openvswitch.org;
> > > > blp at ovn.org
> > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to
> > > > whitelist DPDK physical ports.
> > > >
> > > > Thank you Billy for the review.
> > > > Please find below my reply.
> > > >
> > > > Regards
> > > > _Sugesh
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: O Mahony, Billy
> > > > > Sent: Wednesday, December 6, 2017 5:31 PM
> > > > > To: Chandran, Sugesh <sugesh.chandran at intel.com>;
> > > > > dev at openvswitch.org; blp at ovn.org
> > > > > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option
> > > > > to whitelist DPDK physical ports.
> > > > >
> > > > > Hi Sugesh,
> > > > >
> > > > > This is definitely a very useful feature. I'm looking forward to
> > > > > running trex on the same DUT as my ovs-dpdk.
> [Mooney, Sean K]  you can all ready to this you just need to set the whitelist In
> other_config:dpdk-extra just repeat "-w $address" for each device.
> To have two dpdk primary processes on the same system you will also need to
> change The hugepage prfix used be dpdk which you can also do via the dpdk-
> extra option.
> 
> After this patch we will still be able to specify the whitelist using
> other_config:dpdk-extra correct? If not this may break ovs-dpdk support in
> openstack installers. I ported our whitelist code in networking-ovs-dpdk to use
> dpdk-extra when when we moved the dpdk params to the db and I also added it
> to kolla.
> im pretty sure tripple0 and fule also do the same.
> 
> > > > >
> > > > > However I'd suggest adding an sscanf or some such to verify that
> > > > > the domain is also specified for each whitelist member. And
> > either
> > > > > add the default of '0000' or complain loudly if the domain is
> > absent.
> > > > [Sugesh] Will throw an error in that case then .
> > > >
> > > > >
> > > > > Currently (without this patch) you must specify the domain when
> > > > > adding
> > > ports:
> > > > >    Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else
> > an
> > > > > error such as 'Cannot find unplugged device (05:00.0)'  is
> > reported.
> > > > >
> > > > > And with the patch if you include the domain in the other_config
> > (e.g.
> > > > > other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything
> > > > > works just as before.
> > > > >
> > > > > However with the patch if you add the whitelist *without* a
> > domain e.g.
> > > > > 	ovs-vsctl --no-wait set Open_vSwitch .
> > > > > other_config:dpdk-whitelist-pci- ids="05:00.0"
> > > > >
> > > > > There is no immediate error. However later when doing add-port
> > > > > if you include the domain (current required practice) you will
> > > > > get
> > an error.
> > > > > If you omit the domain all is well.
> > > > [Sugesh] It looks to me, the dpdk-devargs need the PCI id with the
> > '0000'.
> > > > But to bind and PCI scan its not necessary.
> > > > So to keep it consistent, I would add check for PCI-ID in
> > > > whitelist config too, and throw error incase pci-id are mentioned
> > > > wrong(means without
> > > '0000'.
> [Mooney, Sean K] don't assume it is '0000' it is only '0000' if the pci device is A
> child of the first pci root usally connected to socket 0 unless you server is old
> Enough to still use a FSB instead of qpi/dmi.
> 
> > > > Does it looks OK to you?
> > >
> > > [[BO'M]] I think the error is the right thing to do. It would be
> > > tempting to insert the default '0000' if the domain is omitted but
> > > then you would have a confusing inconsistency in that it would be ok
> > > to omit the domain in one place (whitelist) but not in the other
> > (add-port).
> > >
> > [Sugesh] Thank you Billy,Will add the check and error out accordingly.
> > Will release the next patch version by incorporating the comments,
> > once you have complete the review of second part of the patch too.
> [Mooney, Sean K] it should be noted that the dpdk devbing script will only work
> without The domain if its unambiguious. On a multi socket system with the right
> or wrong depending on your perspective placement of physical pci devise it
> possible to get colltions in which case dpdk-devbind will not be able to correctly
> determing what device to manage so you should always use the full pci address.
> 
> > > > >
> > > > > It's a little bit strange as regardless of domain or no domain
> > > > > in the other_config the PCI probe always reports the NIC as
> > expected:
> > > > >     2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device
> > > > > 0000:05:00.0 on NUMA socket -1
> > > > >     2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL:   Invalid NUMA
> > socket,
> > > > > default to 0
> > > > >     2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL:   probe driver:
> > 8086:1572
> > > > > net_i40e
> > > > >
> > > > > I'll be using the other patch in this series "isolate
> > > > > rte-mempool allocation" over the next few days so I'll review
> > > > > that in due
> > course.
> > > > >
> > > > > Thanks,
> > > > > Billy.
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
> > > > > > bounces at openvswitch.org] On Behalf Of Sugesh Chandran
> > > > > > Sent: Friday, November 10, 2017 1:29 AM
> > > > > > To: dev at openvswitch.org; blp at ovn.org
> > > > > > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to
> > > > > > whitelist DPDK physical ports.
> > > > > >
> > > > > > Adding a OVS configuration option to whitelist DPDK physical
> > ports.
> > > > > > By default running multiple instances of DPDK on a single
> > > > > > platform cannot use physical ports at the same time even
> > > > > > though
> > they are distinct.
> > > > > >
> > > > > > The eal init scans all the ports that are bound to DPDK and
> > > > > > initialize the drivers accordingly. This happens for every
> > > > > > DPDK
> > process init.
> > > > > > On a multi instance deployment usecase, it causes issues for
> > > > > > using physical NIC ports.
> > > > > > Consider a two DPDK process that are running on a single
> > > > > > platform, the second DPDK primary process will try to
> > initialize
> > > > > > the drivers for all the physical ports even though it may be
> > used in first DPDK process.
> > > > > >
> > > > > > To avoid this situation user can whitelist the ports for each
> > > > > > DPDK
> > > application.
> > > > > > Whitelisting of ports/PCI-ID in a DPDK process will limit the
> > > > > > eal-init only on those ports.
> > > > > >
> > > > > > To whitelist two physical ports "0000:06:00.0" and
> > > > > > "0000:06:00.1", the configuration option in OVS would be
> > > > > >   ovs-vsctl  set Open_vSwitch . other_config:dpdk-whitelist-
> > pci-
> > > > > > ids="0000:06:00.0,0000:06:00.1"
> > > > > >
> > > > > > To update the whitelist ports, OVS daemon has to be restarted.
> > > > > >
> > > > > > Signed-off-by: Sugesh Chandran <sugesh.chandran at intel.com>
> > > > > > ---
> > > > > >  lib/dpdk.c           | 29 +++++++++++++++++++++++++++++
> > [snip]
> > > > > >
> > > > > > _______________________________________________
> > > > > > dev mailing list
> > > > > > dev at openvswitch.org
> > > > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev


More information about the dev mailing list