[ovs-dev] [PATCH 2/2] Adding configuration option to whitelist DPDK physical ports.
O Mahony, Billy
billy.o.mahony at intel.com
Wed Dec 6 17:30:53 UTC 2017
Hi Sugesh,
This is definitely a very useful feature. I'm looking forward to running trex on the same DUT as my ovs-dpdk.
However I'd suggest adding an sscanf or some such to verify that the domain is also specified for each whitelist member. And either add the default of '0000' or complain loudly if the domain is absent.
Currently (without this patch) you must specify the domain when adding ports:
Vsctl add-port ... options:dpdk-devargs=0000:05:00.0
Or else an error such as 'Cannot find unplugged device (05:00.0)' is reported.
And with the patch if you include the domain in the other_config (e.g. other_config:dpdk-whitelist-pci-ids="0000:05:00.0") everything works just as before.
However with the patch if you add the whitelist *without* a domain e.g.
ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-whitelist-pci-ids="05:00.0"
There is no immediate error. However later when doing add-port if you include the domain (current required practice) you will get an error. If you omit the domain all is well.
It's a little bit strange as regardless of domain or no domain in the other_config the PCI probe always reports the NIC as expected:
2017-12-06T16:55:27Z|00013|dpdk|INFO|EAL: PCI device 0000:05:00.0 on NUMA socket -1
2017-12-06T16:55:27Z|00014|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
2017-12-06T16:55:27Z|00015|dpdk|INFO|EAL: probe driver: 8086:1572 net_i40e
I'll be using the other patch in this series "isolate rte-mempool allocation" over the next few days so I'll review that in due course.
Thanks,
Billy.
> -----Original Message-----
> From: ovs-dev-bounces at openvswitch.org [mailto:ovs-dev-
> bounces at openvswitch.org] On Behalf Of Sugesh Chandran
> Sent: Friday, November 10, 2017 1:29 AM
> To: dev at openvswitch.org; blp at ovn.org
> Subject: [ovs-dev] [PATCH 2/2] Adding configuration option to whitelist DPDK
> physical ports.
>
> Adding a OVS configuration option to whitelist DPDK physical ports. By default
> running multiple instances of DPDK on a single platform cannot use physical
> ports at the same time even though they are distinct.
>
> The eal init scans all the ports that are bound to DPDK and initialize the drivers
> accordingly. This happens for every DPDK process init.
> On a multi instance deployment usecase, it causes issues for using physical NIC
> ports.
> Consider a two DPDK process that are running on a single platform, the second
> DPDK primary process will try to initialize the drivers for all the physical ports
> even though it may be used in first DPDK process.
>
> To avoid this situation user can whitelist the ports for each DPDK application.
> Whitelisting of ports/PCI-ID in a DPDK process will limit the eal-init only on those
> ports.
>
> To whitelist two physical ports "0000:06:00.0" and "0000:06:00.1", the
> configuration option in OVS would be
> ovs-vsctl set Open_vSwitch . other_config:dpdk-whitelist-pci-
> ids="0000:06:00.0,0000:06:00.1"
>
> To update the whitelist ports, OVS daemon has to be restarted.
>
> Signed-off-by: Sugesh Chandran <sugesh.chandran at intel.com>
> ---
> lib/dpdk.c | 29 +++++++++++++++++++++++++++++
> vswitchd/vswitch.xml | 21 +++++++++++++++++++++
> 2 files changed, 50 insertions(+)
>
> diff --git a/lib/dpdk.c b/lib/dpdk.c
> index 9d187c7..0f11977 100644
> --- a/lib/dpdk.c
> +++ b/lib/dpdk.c
> @@ -323,6 +323,34 @@ dpdk_isolate_rte_mem_config(const struct smap
> *ovs_other_config, }
>
> static void
> +dpdk_whitelist_pci_ids(const struct smap *ovs_other_config, char ***argv,
> + int *argc)
> +{
> + const char *pci_ids;
> + char *pci_dev;
> + int len;
> + int i;
> + pci_ids = smap_get(ovs_other_config, "dpdk-whitelist-pci-ids");
> + if (!pci_ids) {
> + return;
> + }
> + len = strlen(pci_ids);
> + do {
> + i = strcspn(pci_ids, ",");
> + pci_dev = xmemdup0(pci_ids, i);
> + if (!strlen(pci_dev)) {
> + break;
> + }
> + *argv = grow_argv(argv, *argc, 2);
> + (*argv)[(*argc)++] = xstrdup("-w");
> + (*argv)[(*argc)++] = pci_dev;
> + i++;
> + pci_ids += i;
> + len -= i;
> + } while (pci_ids && len > 0);
> +}
> +
> +static void
> dpdk_init__(const struct smap *ovs_other_config) {
> char **argv = NULL, **argv_to_release = NULL; @@ -409,6 +437,7 @@
> dpdk_init__(const struct smap *ovs_other_config)
> }
>
> dpdk_isolate_rte_mem_config(ovs_other_config, &argv, &argc);
> + dpdk_whitelist_pci_ids(ovs_other_config, &argv, &argc);
> argv = grow_argv(&argv, argc, 1);
> argv[argc] = NULL;
>
> diff --git a/vswitchd/vswitch.xml b/vswitchd/vswitch.xml index
> 7462b30..0b64b25 100644
> --- a/vswitchd/vswitch.xml
> +++ b/vswitchd/vswitch.xml
> @@ -442,6 +442,27 @@
> </p>
> </column>
>
> + <column name="other_config" key="dpdk-whitelist-pci-ids">
> + <p>
> + Specifies list of pci-ids separated by , for whitelisting available
> + physical NIC ports in OVS. The option valid only when DPDK is enabled
> + in OVS and the ports are already bound to DPDK userspace driver.
> + </p>
> + <p>
> + By default, all the DPDK bound ports are initialized at the time of
> + vswitchd start. Whitelisting a list of pci-ids is used to limit the
> + initialization only to specific ports. Changing this value requires
> + restarting the daemon.
> + </p>
> + <p>
> + This option allow user to run multiple instance of DPDK including
> + vswitchd simultaneously by exclusively allocating the physical NIC
> + ports between them. Its impossible to use the NIC ports in
> + multiple primary DPDK processes without whitelisting them
> + even the application is using distinct ports.
> + </p>
> + </column>
> +
> </group>
>
> <group title="Status">
> --
> 2.7.4
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
More information about the dev
mailing list