[ovs-dev] [PATCH v5] netdev-dpdk: add hotplug support

Stokes, Ian ian.stokes at intel.com
Mon May 16 17:13:38 UTC 2016


> -----Original Message-----
> From: Mauricio Vasquez B
> [mailto:mauricio.vasquezbernal at studenti.polito.it]
> Sent: Friday, May 13, 2016 9:26 PM
> To: dev at openvswitch.org
> Cc: aconole at redhat.com; lw at cn.fujitsu.com; mchandras at suse.de; Stokes,
> Ian; fbl at sysclose.org; Traynor, Kevin; diproiettod at vmware.com
> Subject: [PATCH v5] netdev-dpdk: add hotplug support
> 
> In order to use dpdk ports in ovs they have to be bound to a DPDK
> compatible driver before ovs is started.
> 
> This patch adds the possibility to hotplug (or hot-unplug) a device
> after ovs has been started. The implementation adds two appctl commands:
> netdev-dpdk/port-attach and netdev-dpdk/port-detach
> 
> After the user attaches a new device, it has to be added to a bridge
> using the add-port command, similarly, before detaching a device, it has
> to be removed using the del-port command.
> 
> Signed-off-by: Mauricio Vasquez B
> <mauricio.vasquezbernal at studenti.polito.it>
> ---
> v5:
>  - use two appctl commands instead of a single one
>  - rebase to master
> v4:
>  - fix typo in commit message
>  - remove unnecessary whitespace change in INSTALL.DPDK.md
> v3:
>  - create dpdk_port_attach and dpdk_port_detach functions
>  - modify mutex locking order
> v2:
>  - use rte_eth_dev_is_valid_port() to check if a port is valid
>  INSTALL.DPDK.md   |  24 +++++++++++++
>  NEWS              |   1 +
>  lib/netdev-dpdk.c | 101
> ++++++++++++++++++++++++++++++++++++++++++++++++++----
>  3 files changed, 119 insertions(+), 7 deletions(-)
> 
> diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md index 93f92e4..8d5a0e0
> 100644
> --- a/INSTALL.DPDK.md
> +++ b/INSTALL.DPDK.md
> @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd:
>     For more details regarding egress-policer parameters please refer to
> the
>     vswitch.xml.
> 
> +9. Port Hotplug
> +
> +   Ovs supports port hotplugging, it allows to use ports that were not
> bound
> +   to DPDK when vswitchd was started.
> +   In order to attach a port, it has to be bound to DPDK using the
> +   dpdk_nic_bind.py script:
> +
> +   `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0`
> +
> +   Then it can be attached to OVS:
> +
> +   `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0`
> +
> +   At this point, the user can create a ovs port using the add-port
> command.
> +
> +   It is also possible to detach a port from ovs, the user has to
> remove the
> +   port using the del-port command, then it can be detached using:
> +
> +   `ovs-appctl netdev-dpdk/port-detach dpdk0`
> +
> +   This feature is not supported by all the NICs, please refer to the
> +   [DPDK Port Hotplug Framework] in order to get more information.
> +
>  Performance Tuning:
>  -------------------
> 
> @@ -999,3 +1022,4 @@ Please report problems to bugs at openvswitch.org.
>  [INSTALL.md]:INSTALL.md
>  [DPDK Linux GSG]:
> http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-
> unbinding-network-ports-to-from-the-igb-uioor-vfio-modules
>  [DPDK Docs]: http://dpdk.org/doc
> +[DPDK Port Hotplug Framework]:
> +http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html
> diff --git a/NEWS b/NEWS
> index 4e81cad..d89d9a7 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -32,6 +32,7 @@ Post-v2.5.0
>       * DB entries have been added for many of the DPDK EAL command line
>         arguments. Additional arguments can be passed via the dpdk-extra
>         entry.
> +     * Port Hotplug is now supported.
>     - ovs-benchmark: This utility has been removed due to lack of use
> and
>       bitrot.
>     - ovs-appctl:
> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c index
> af86d19..6159a60 100644
> --- a/lib/netdev-dpdk.c
> +++ b/lib/netdev-dpdk.c
> @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev)
> OVS_REQUIRES(dpdk_mutex)
>      int diag;
>      int n_rxq, n_txq;
> 
> -    if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) {
> +    if (!rte_eth_dev_is_valid_port(dev->port_id)) {
>          return ENODEV;
>      }
> 
> @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn
> *conn, int argc,
>      unixctl_command_reply(conn, "OK");
>  }
> 
> +static void
> +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED,
> +                        const char *argv[], void *aux OVS_UNUSED) {
> +    int ret;
> +    char response[128];
> +    uint8_t port_id;
> +
> +    ovs_mutex_lock(&dpdk_mutex);
> +
> +    ret = rte_eth_dev_attach(argv[1], &port_id);
> +    if (ret < 0) {
> +        snprintf(response, sizeof(response),
> +                 "Error attaching device '%s'", argv[1]);
> +        ovs_mutex_unlock(&dpdk_mutex);
> +        unixctl_command_reply_error(conn, response);
> +        return;
> +    }
> +
> +    snprintf(response, sizeof(response),
> +             "Device '%s' has been attached as 'dpdk%d'", argv[1],
> + port_id);
> +
> +    ovs_mutex_unlock(&dpdk_mutex);
> +    unixctl_command_reply(conn, response); }
> +
> +static void
> +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED,
> +                        const char *argv[], void *aux OVS_UNUSED) {
> +    int ret;
> +    char response[128];
> +    unsigned int parsed_port;
> +    uint8_t port_id;
> +    char devname[RTE_ETH_NAME_MAX_LEN];
> +
> +    ovs_mutex_lock(&dpdk_mutex);
> +
> +    ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port);
> +    if (ret) {
> +        snprintf(response, sizeof(response),
> +                 "'%s' is not a valid port", argv[1]);
> +        goto error;
> +    }
> +
> +    port_id = parsed_port;
> +
> +    struct netdev *netdev = netdev_from_name(argv[1]);
> +    if (netdev) {
> +        netdev_close(netdev);
> +        snprintf(response, sizeof(response),
> +                 "Port '%s' is being used. Remove it before detaching",
> +                 argv[1]);
> +        goto error;
> +    }
> +
> +    rte_eth_dev_close(port_id);
> +
> +    ret = rte_eth_dev_detach(port_id, devname);
> +    if (ret < 0) {
> +        snprintf(response, sizeof(response),
> +                 "Port '%s' can not be detached", argv[1]);
> +        goto error;
> +    }
> +
> +    snprintf(response, sizeof(response),
> +             "Port '%s' has been detached", argv[1]);
> +
> +    ovs_mutex_unlock(&dpdk_mutex);
> +    unixctl_command_reply(conn, response);
> +    return;
> +
> +error:
> +    ovs_mutex_unlock(&dpdk_mutex);
> +    unixctl_command_reply_error(conn, response); }
> +
>  /*
>   * Set virtqueue flags so that we do not receive interrupts.
>   */
> @@ -2404,6 +2481,15 @@ dpdk_common_init(void)
>                               "[netdev] up|down", 1, 2,
>                               netdev_dpdk_set_admin_state, NULL);
> 
> +    unixctl_command_register("netdev-dpdk/port-attach",
> +                             "pci address of device", 1, 1,
> +                             netdev_dpdk_port_attach, NULL);
> +
> +    unixctl_command_register("netdev-dpdk/port-detach",
> +                             "port", 1, 1,
> +                             netdev_dpdk_port_detach, NULL);
> +
> +    ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL);
>  }
> 
>  /* Client Rings */
> @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned
> int port_no,  {
>      struct dpdk_ring *ivshmem;
>      char ring_name[RTE_RING_NAMESIZE];
> -    int err;
> +    int err, port_id;
> 
>      ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem);
>      if (ivshmem == NULL) {
> @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[], unsigned
> int port_no,
>          return ENOMEM;
>      }
> 
> -    err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1,
> -                             &ivshmem->cring_tx, 1, SOCKET0);
> +    port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1,
> +                                 &ivshmem->cring_tx, 1, SOCKET0);
> 
> -    if (err < 0) {
> +    if (port_id < 0) {
>          rte_free(ivshmem);
>          return ENODEV;
>      }
> 
>      ivshmem->user_port_id = port_no;
> -    ivshmem->eth_port_id = rte_eth_dev_count() - 1;
> +    ivshmem->eth_port_id = port_id;
> +    *eth_port_id = port_id;
> +
>      ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node);
> 
> -    *eth_port_id = ivshmem->eth_port_id;
>      return 0;
>  }
> 
> --
> 1.9.1

Thanks for the latest patch revision Mauricio.
I've tested it with an Intel 82599 NIC and it works without issue.

However I did come across an issue related to the XL710 interface when it is attached, detached and then re-attached, essentially the rte_init function fails as the device appears as busy. I think this issue is specific to the Xl710 however as the same behavior could not be replicated with the Intel 82599 NIC.

As you have it called out in your notes that hot plug support is interface dependent I guess this is ok. It's seems to be a DPDK issue for the XL710 and will need further investigation separate to this work.

Tested-by: ian.stokes at intel.com

Thanks
Ian



More information about the dev mailing list