[ovs-dev] [PATCH 2/2] Adding configuration option to whitelist DPDK physical ports.

O Mahony, Billy billy.o.mahony at intel.com
Thu Dec 7 11:46:43 UTC 2017


Hi Sugesh,

> -----Original Message-----
> From: Chandran, Sugesh
> Sent: Wednesday, December 6, 2017 6:23 PM
> To: O Mahony, Billy <billy.o.mahony at intel.com>; dev at openvswitch.org;
> blp at ovn.org
> Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist
> DPDK physical ports.
> 
> Thank you Billy for the review.
> Please find below my reply.
> 
> Regards
> _Sugesh
> 
> 
> > -----Original Message-----
> > From: O Mahony, Billy
> > Sent: Wednesday, December 6, 2017 5:31 PM
> > To: Chandran, Sugesh <sugesh.chandran at intel.com>; dev at openvswitch.org;
> > blp at ovn.org
> > Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to
> > whitelist DPDK physical ports.
> >
> > Hi Sugesh,
> >
> > This is definitely a very useful feature. I'm looking forward to
> > running trex on the same DUT as my ovs-dpdk.
> >
> > However I'd suggest adding an sscanf or some such to verify that the
> > domain is also specified for each whitelist member. And either add the
> > default of '0000' or complain loudly if the domain is absent.
> [Sugesh] Will throw an error in that case then .
> 
> >
> > Currently (without this patch) you must specify the domain when adding ports:
> >    Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else an
> > error such as 'Cannot find unplugged device (05:00.0)'  is reported.
> >
> > And with the patch if you include the domain in the other_config (e.g.
> > other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything works
> > just as before.
> >
> > However with the patch if you add the whitelist *without* a domain e.g.
> > 	ovs-vsctl --no-wait set Open_vSwitch .
> > other_config:dpdk-whitelist-pci- ids="05:00.0"
> >
> > There is no immediate error. However later when doing add-port if you
> > include the domain (current required practice) you will get an error.
> > If you omit the domain all is well.
> [Sugesh] It looks to me, the dpdk-devargs need the PCI id with the '0000'.
> But to bind and PCI scan its not necessary.
> So to keep it consistent, I would add check for PCI-ID in whitelist config too, and
> throw error incase pci-id are mentioned wrong(means without '0000'.
> Does it looks OK to you?

[[BO'M]] I think the error is the right thing to do. It would be tempting to insert the default '0000' if the domain is omitted but then you would have a confusing inconsistency in that it would be ok to omit the domain in one place (whitelist) but not in the other (add-port).

> >
> > It's a little bit strange as regardless of domain or no domain in the
> > other_config the PCI probe always reports the NIC as expected:
> >     2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device 0000:05:00.0
> > on NUMA socket -1
> >     2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL:   Invalid NUMA socket,
> > default to 0
> >     2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL:   probe driver: 8086:1572
> > net_i40e
> >
> > I'll be using the other patch in this series "isolate rte-mempool
> > allocation" over the next few days so I'll review that in due course.
> >
> > Thanks,
> > Billy.
> >
> > > -----Original Message-----
> > > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
> > > bounces at openvswitch.org] On Behalf Of Sugesh Chandran
> > > Sent: Friday, November 10, 2017 1:29 AM
> > > To: dev at openvswitch.org; blp at ovn.org
> > > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to
> > > whitelist DPDK physical ports.
> > >
> > > Adding a OVS configuration option to whitelist DPDK physical ports.
> > > By default running multiple instances of DPDK on a single platform
> > > cannot use physical ports at the same time even though they are distinct.
> > >
> > > The eal init scans all the ports that are bound to DPDK and
> > > initialize the drivers accordingly. This happens for every DPDK process init.
> > > On a multi instance deployment usecase, it causes issues for using
> > > physical NIC ports.
> > > Consider a two DPDK process that are running on a single platform,
> > > the second DPDK primary process will try to initialize the drivers
> > > for all the physical ports even though it may be used in first DPDK process.
> > >
> > > To avoid this situation user can whitelist the ports for each DPDK application.
> > > Whitelisting of ports/PCI-ID in a DPDK process will limit the
> > > eal-init only on those ports.
> > >
> > > To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1",
> > > the configuration option in OVS would be
> > >   ovs-vsctl  set Open_vSwitch . other_config:dpdk-whitelist-pci-
> > > ids="0000:06:00.0,0000:06:00.1"
> > >
> > > To update the whitelist ports, OVS daemon has to be restarted.
> > >
> > > Signed-off-by: Sugesh Chandran <sugesh.chandran at intel.com>
> > > ---
> > >  lib/dpdk.c           | 29 +++++++++++++++++++++++++++++
> > >  vswitchd/vswitch.xml | 21 +++++++++++++++++++++
> > >  2 files changed, 50 insertions(+)
> > >
> > > diff --git a/lib/dpdk.c b/lib/dpdk.c index 9d187c7..0f11977 100644
> > > --- a/lib/dpdk.c
> > > +++ b/lib/dpdk.c
> > > @@ -323,6 +323,34 @@ dpdk_isolate_rte_mem_config(const struct smap
> > > *ovs_other_config,  }
> > >
> > >  static void
> > > +dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv,
> > > +                       int *argc)
> > > +{
> > > +    const char *pci_ids;
> > > +    char *pci_dev;
> > > +    int len;
> > > +    int i;
> > > +    pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids");
> > > +    if (!pci_ids) {
> > > +        return;
> > > +    }
> > > +    len = strlen(pci_ids);
> > > +    do {
> > > +        i = strcspn(pci_ids, ",");
> > > +        pci_dev = xmemdup0(pci_ids, i);
> > > +        if (!strlen(pci_dev)) {
> > > +             break;
> > > +        }
> > > +        *argv = grow_argv(argv, *argc, 2);
> > > +        (*argv)[(*argc)++] = xstrdup("-w");
> > > +        (*argv)[(*argc)++] = pci_dev;
> > > +        i++;
> > > +        pci_ids += i;
> > > +        len -= i;
> > > +    } while (pci_ids && len > 0);
> > > +}
> > > +
> > > +static void
> > >  dpdk_init__(const struct smap *ovs_other_config)  {
> > >      char **argv = NULL, **argv_to_release = NULL; @@ -409,6 +437,7
> > > @@ dpdk_init__(const struct smap *ovs_other_config)
> > >      }
> > >
> > >      dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc);
> > > +    dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc);
> > >      argv = grow_argv(&argv, argc, 1);
> > >      argv[argc] = NULL;
> > >
> > > diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index
> > > 7462b30..0b64b25 100644
> > > --- a/vswitchd/vswitch.xml
> > > +++ b/vswitchd/vswitch.xml
> > > @@ -442,6 +442,27 @@
> > >          </p>
> > >        </column>
> > >
> > > +      <column name="other_config" key="dpdk-whitelist-pci-ids">
> > > +        <p>
> > > +          Specifies list of pci-ids separated by , for whitelisting available
> > > +          physical NIC ports in OVS. The option valid only when DPDK is enabled
> > > +          in OVS and the ports are already bound to DPDK userspace driver.
> > > +        </p>
> > > +        <p>
> > > +          By default, all the DPDK bound ports are initialized at the time of
> > > +          vswitchd start. Whitelisting a list of pci-ids is used to limit the
> > > +          initialization only to specific ports. Changing this value requires
> > > +          restarting the daemon.
> > > +        </p>
> > > +        <p>
> > > +         This option allow user to run multiple instance of DPDK including
> > > +         vswitchd simultaneously by exclusively allocating the physical NIC
> > > +         ports between them. Its impossible to use the NIC ports in
> > > +         multiple primary DPDK processes without whitelisting them
> > > +         even the application is using distinct ports.
> > > +        </p>
> > > +      </column>
> > > +
> > >      </group>
> > >
> > >      <group title="Status">
> > > --
> > > 2.7.4
> > >
> > > _______________________________________________
> > > dev mailing list
> > > dev at openvswitch.org
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev


More information about the dev mailing list