[ovs-dev] [PATCH 2/2] Adding configuration option to whitelist DPDK physical ports.

Chandran, Sugesh sugesh.chandran at intel.com
Wed Dec 6 18:23:12 UTC 2017


Thank you Billy for the review. 
Please find below my reply.

Regards
_Sugesh


> -----Original Message-----
> From: O Mahony, Billy
> Sent: Wednesday, December 6, 2017 5:31 PM
> To: Chandran, Sugesh <sugesh.chandran at intel.com>; dev at openvswitch.org;
> blp at ovn.org
> Subject: RE: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist
> DPDK physical ports.
> 
> Hi Sugesh,
> 
> This is definitely a very useful feature. I'm looking forward to running trex on the
> same DUT as my ovs-dpdk.
> 
> However I'd suggest adding an sscanf or some such to verify that the domain is
> also specified for each whitelist member. And either add the default of '0000' or
> complain loudly if the domain is absent.
[Sugesh] Will throw an error in that case then .

> 
> Currently (without this patch) you must specify the domain when adding ports:
>    Vsctl add-port ... options:dpdk-devargs=0000:05:00.0 Or else an error such as
> 'Cannot find unplugged device (05:00.0)'  is reported.
> 
> And with the patch if you include the domain in the other_config (e.g.
> other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything works just as
> before.
> 
> However with the patch if you add the whitelist *without* a domain e.g.
> 	ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-whitelist-pci-
> ids="05:00.0"
> 
> There is no immediate error. However later when doing add-port if you include
> the domain (current required practice) you will get an error. If you omit the
> domain all is well.
[Sugesh] It looks to me, the dpdk-devargs need the PCI id with the '0000'.
But to bind and PCI scan its not necessary. 
So to keep it consistent, I would add check for PCI-ID in whitelist config too, 
and throw error incase pci-id are mentioned wrong(means without '0000'.
Does it looks OK to you?
> 
> It's a little bit strange as regardless of domain or no domain in the other_config
> the PCI probe always reports the NIC as expected:
>     2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device 0000:05:00.0 on
> NUMA socket -1
>     2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL:   Invalid NUMA socket,
> default to 0
>     2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL:   probe driver: 8086:1572
> net_i40e
> 
> I'll be using the other patch in this series "isolate rte-mempool allocation" over
> the next few days so I'll review that in due course.
> 
> Thanks,
> Billy.
> 
> > -----Original Message-----
> > From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
> > bounces at openvswitch.org] On Behalf Of Sugesh Chandran
> > Sent: Friday, November 10, 2017 1:29 AM
> > To: dev at openvswitch.org; blp at ovn.org
> > Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to
> > whitelist DPDK physical ports.
> >
> > Adding a OVS configuration option to whitelist DPDK physical ports. By
> > default running multiple instances of DPDK on a single platform cannot
> > use physical ports at the same time even though they are distinct.
> >
> > The eal init scans all the ports that are bound to DPDK and initialize
> > the drivers accordingly. This happens for every DPDK process init.
> > On a multi instance deployment usecase, it causes issues for using
> > physical NIC ports.
> > Consider a two DPDK process that are running on a single platform, the
> > second DPDK primary process will try to initialize the drivers for all
> > the physical ports even though it may be used in first DPDK process.
> >
> > To avoid this situation user can whitelist the ports for each DPDK application.
> > Whitelisting of ports/PCI-ID in a DPDK process will limit the eal-init
> > only on those ports.
> >
> > To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", the
> > configuration option in OVS would be
> >   ovs-vsctl  set Open_vSwitch . other_config:dpdk-whitelist-pci-
> > ids="0000:06:00.0,0000:06:00.1"
> >
> > To update the whitelist ports, OVS daemon has to be restarted.
> >
> > Signed-off-by: Sugesh Chandran <sugesh.chandran at intel.com>
> > ---
> >  lib/dpdk.c           | 29 +++++++++++++++++++++++++++++
> >  vswitchd/vswitch.xml | 21 +++++++++++++++++++++
> >  2 files changed, 50 insertions(+)
> >
> > diff --git a/lib/dpdk.c b/lib/dpdk.c
> > index 9d187c7..0f11977 100644
> > --- a/lib/dpdk.c
> > +++ b/lib/dpdk.c
> > @@ -323,6 +323,34 @@ dpdk_isolate_rte_mem_config(const struct smap
> > *ovs_other_config,  }
> >
> >  static void
> > +dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv,
> > +                       int *argc)
> > +{
> > +    const char *pci_ids;
> > +    char *pci_dev;
> > +    int len;
> > +    int i;
> > +    pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids");
> > +    if (!pci_ids) {
> > +        return;
> > +    }
> > +    len = strlen(pci_ids);
> > +    do {
> > +        i = strcspn(pci_ids, ",");
> > +        pci_dev = xmemdup0(pci_ids, i);
> > +        if (!strlen(pci_dev)) {
> > +             break;
> > +        }
> > +        *argv = grow_argv(argv, *argc, 2);
> > +        (*argv)[(*argc)++] = xstrdup("-w");
> > +        (*argv)[(*argc)++] = pci_dev;
> > +        i++;
> > +        pci_ids += i;
> > +        len -= i;
> > +    } while (pci_ids && len > 0);
> > +}
> > +
> > +static void
> >  dpdk_init__(const struct smap *ovs_other_config)  {
> >      char **argv = NULL, **argv_to_release = NULL; @@ -409,6 +437,7 @@
> > dpdk_init__(const struct smap *ovs_other_config)
> >      }
> >
> >      dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc);
> > +    dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc);
> >      argv = grow_argv(&argv, argc, 1);
> >      argv[argc] = NULL;
> >
> > diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index
> > 7462b30..0b64b25 100644
> > --- a/vswitchd/vswitch.xml
> > +++ b/vswitchd/vswitch.xml
> > @@ -442,6 +442,27 @@
> >          </p>
> >        </column>
> >
> > +      <column name="other_config" key="dpdk-whitelist-pci-ids">
> > +        <p>
> > +          Specifies list of pci-ids separated by , for whitelisting available
> > +          physical NIC ports in OVS. The option valid only when DPDK is enabled
> > +          in OVS and the ports are already bound to DPDK userspace driver.
> > +        </p>
> > +        <p>
> > +          By default, all the DPDK bound ports are initialized at the time of
> > +          vswitchd start. Whitelisting a list of pci-ids is used to limit the
> > +          initialization only to specific ports. Changing this value requires
> > +          restarting the daemon.
> > +        </p>
> > +        <p>
> > +         This option allow user to run multiple instance of DPDK including
> > +         vswitchd simultaneously by exclusively allocating the physical NIC
> > +         ports between them. Its impossible to use the NIC ports in
> > +         multiple primary DPDK processes without whitelisting them
> > +         even the application is using distinct ports.
> > +        </p>
> > +      </column>
> > +
> >      </group>
> >
> >      <group title="Status">
> > --
> > 2.7.4
> >
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev


More information about the dev mailing list