[ovs-dev] [PATCH v5] netdev-dpdk: add hotplug support
Mauricio Vásquez
mauricio.vasquezbernal at studenti.polito.it
Thu May 26 21:30:16 UTC 2016
Hi,
I just sent v6: http://openvswitch.org/pipermail/dev/2016-May/071786.html
On Thu, May 19, 2016 at 9:30 AM, Mauricio Vásquez <
mauricio.vasquezbernal at studenti.polito.it> wrote:
>
>
> On Tue, May 17, 2016 at 12:44 AM, Flavio Leitner <fbl at sysclose.org> wrote:
>
>> On Fri, May 13, 2016 at 10:25:41PM +0200, Mauricio Vasquez B wrote:
>> > In order to use dpdk ports in ovs they have to be bound to a DPDK
>> > compatible driver before ovs is started.
>> >
>> > This patch adds the possibility to hotplug (or hot-unplug) a device
>> > after ovs has been started. The implementation adds two appctl commands:
>> > netdev-dpdk/port-attach and netdev-dpdk/port-detach
>> >
>> > After the user attaches a new device, it has to be added to a bridge
>> > using the add-port command, similarly, before detaching a device,
>> > it has to be removed using the del-port command.
>> >
>> > Signed-off-by: Mauricio Vasquez B <
>> mauricio.vasquezbernal at studenti.polito.it>
>>
>> The patch looks good, but it didn't work for me with VFIO (it's
>> not supported by the DPDK hotplug framework). Since the doc
>> includes instructions to use VFIO, should we add a note for the
>> lack of hotplug support?
>>
>>
> Yes, we should. I'll wait few days to see if there is more feedback and
> I'll send a new version including that note.
>
>
>> The attach/detach operations seem to work with uio_pci_generic,
>> but it doesn't move packets. The interface statistics shows
>> only 'errs' increasing a lot and few 'drops'.
>>
>> Kernel: 4.5.0+
>> OVS: master (b396293a) + patch
>> DPDK: 16.04
>> Drivers: vfio-pci or uio_pci_generic
>> Hardware: 82599ES
>>
>>
> I tried uio_pci_generic in the following conditions without any problem.
>
> Kernel: 3.19.0-32-generic
> OVS: master(d640a7e1) + patch
> DPDK: 16.04
> Hardware: X540-AT2
>
>
>> Do you know if there are patches/plans to support that with VFIO or
>> uio_pci_generic?
>>
>
> I am not aware of any plans for it.
>
> Thanks for reviewing Flavio!
>
>
>> fbl
>>
>>
>> > ---
>> > v5:
>> > - use two appctl commands instead of a single one
>> > - rebase to master
>> > v4:
>> > - fix typo in commit message
>> > - remove unnecessary whitespace change in INSTALL.DPDK.md
>> > v3:
>> > - create dpdk_port_attach and dpdk_port_detach functions
>> > - modify mutex locking order
>> > v2:
>> > - use rte_eth_dev_is_valid_port() to check if a port is valid
>> > INSTALL.DPDK.md | 24 +++++++++++++
>> > NEWS | 1 +
>> > lib/netdev-dpdk.c | 101
>> ++++++++++++++++++++++++++++++++++++++++++++++++++----
>> > 3 files changed, 119 insertions(+), 7 deletions(-)
>> >
>> > diff --git a/INSTALL.DPDK.md b/INSTALL.DPDK.md
>> > index 93f92e4..8d5a0e0 100644
>> > --- a/INSTALL.DPDK.md
>> > +++ b/INSTALL.DPDK.md
>> > @@ -267,6 +267,29 @@ Using the DPDK with ovs-vswitchd:
>> > For more details regarding egress-policer parameters please refer
>> to the
>> > vswitch.xml.
>> >
>> > +9. Port Hotplug
>> > +
>> > + Ovs supports port hotplugging, it allows to use ports that were not
>> bound
>> > + to DPDK when vswitchd was started.
>> > + In order to attach a port, it has to be bound to DPDK using the
>> > + dpdk_nic_bind.py script:
>> > +
>> > + `$DPDK_DIR/tools/dpdk_nic_bind.py --bind=igb_uio 0000:01:00.0`
>> > +
>> > + Then it can be attached to OVS:
>> > +
>> > + `ovs-appctl netdev-dpdk/port-attach 0000:01:00.0`
>> > +
>> > + At this point, the user can create a ovs port using the add-port
>> command.
>> > +
>> > + It is also possible to detach a port from ovs, the user has to
>> remove the
>> > + port using the del-port command, then it can be detached using:
>> > +
>> > + `ovs-appctl netdev-dpdk/port-detach dpdk0`
>> > +
>> > + This feature is not supported by all the NICs, please refer to the
>> > + [DPDK Port Hotplug Framework] in order to get more information.
>> > +
>> > Performance Tuning:
>> > -------------------
>> >
>> > @@ -999,3 +1022,4 @@ Please report problems to bugs at openvswitch.org.
>> > [INSTALL.md]:INSTALL.md
>> > [DPDK Linux GSG]:
>> http://www.dpdk.org/doc/guides/linux_gsg/build_dpdk.html#binding-and-unbinding-network-ports-to-from-the-igb-uioor-vfio-modules
>> > [DPDK Docs]: http://dpdk.org/doc
>> > +[DPDK Port Hotplug Framework]:
>> http://dpdk.org/doc/guides/prog_guide/port_hotplug_framework.html
>> > diff --git a/NEWS b/NEWS
>> > index 4e81cad..d89d9a7 100644
>> > --- a/NEWS
>> > +++ b/NEWS
>> > @@ -32,6 +32,7 @@ Post-v2.5.0
>> > * DB entries have been added for many of the DPDK EAL command line
>> > arguments. Additional arguments can be passed via the dpdk-extra
>> > entry.
>> > + * Port Hotplug is now supported.
>> > - ovs-benchmark: This utility has been removed due to lack of use
>> and
>> > bitrot.
>> > - ovs-appctl:
>> > diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
>> > index af86d19..6159a60 100644
>> > --- a/lib/netdev-dpdk.c
>> > +++ b/lib/netdev-dpdk.c
>> > @@ -630,7 +630,7 @@ dpdk_eth_dev_init(struct netdev_dpdk *dev)
>> OVS_REQUIRES(dpdk_mutex)
>> > int diag;
>> > int n_rxq, n_txq;
>> >
>> > - if (dev->port_id < 0 || dev->port_id >= rte_eth_dev_count()) {
>> > + if (!rte_eth_dev_is_valid_port(dev->port_id)) {
>> > return ENODEV;
>> > }
>> >
>> > @@ -2118,6 +2118,83 @@ netdev_dpdk_set_admin_state(struct unixctl_conn
>> *conn, int argc,
>> > unixctl_command_reply(conn, "OK");
>> > }
>> >
>> > +static void
>> > +netdev_dpdk_port_attach(struct unixctl_conn *conn, int argc OVS_UNUSED,
>> > + const char *argv[], void *aux OVS_UNUSED)
>> > +{
>> > + int ret;
>> > + char response[128];
>> > + uint8_t port_id;
>> > +
>> > + ovs_mutex_lock(&dpdk_mutex);
>> > +
>> > + ret = rte_eth_dev_attach(argv[1], &port_id);
>> > + if (ret < 0) {
>> > + snprintf(response, sizeof(response),
>> > + "Error attaching device '%s'", argv[1]);
>> > + ovs_mutex_unlock(&dpdk_mutex);
>> > + unixctl_command_reply_error(conn, response);
>> > + return;
>> > + }
>> > +
>> > + snprintf(response, sizeof(response),
>> > + "Device '%s' has been attached as 'dpdk%d'", argv[1],
>> port_id);
>> > +
>> > + ovs_mutex_unlock(&dpdk_mutex);
>> > + unixctl_command_reply(conn, response);
>> > +}
>> > +
>> > +static void
>> > +netdev_dpdk_port_detach(struct unixctl_conn *conn, int argc OVS_UNUSED,
>> > + const char *argv[], void *aux OVS_UNUSED)
>> > +{
>> > + int ret;
>> > + char response[128];
>> > + unsigned int parsed_port;
>> > + uint8_t port_id;
>> > + char devname[RTE_ETH_NAME_MAX_LEN];
>> > +
>> > + ovs_mutex_lock(&dpdk_mutex);
>> > +
>> > + ret = dpdk_dev_parse_name(argv[1], "dpdk", &parsed_port);
>> > + if (ret) {
>> > + snprintf(response, sizeof(response),
>> > + "'%s' is not a valid port", argv[1]);
>> > + goto error;
>> > + }
>> > +
>> > + port_id = parsed_port;
>> > +
>> > + struct netdev *netdev = netdev_from_name(argv[1]);
>> > + if (netdev) {
>> > + netdev_close(netdev);
>> > + snprintf(response, sizeof(response),
>> > + "Port '%s' is being used. Remove it before detaching",
>> > + argv[1]);
>> > + goto error;
>> > + }
>> > +
>> > + rte_eth_dev_close(port_id);
>> > +
>> > + ret = rte_eth_dev_detach(port_id, devname);
>> > + if (ret < 0) {
>> > + snprintf(response, sizeof(response),
>> > + "Port '%s' can not be detached", argv[1]);
>> > + goto error;
>> > + }
>> > +
>> > + snprintf(response, sizeof(response),
>> > + "Port '%s' has been detached", argv[1]);
>> > +
>> > + ovs_mutex_unlock(&dpdk_mutex);
>> > + unixctl_command_reply(conn, response);
>> > + return;
>> > +
>> > +error:
>> > + ovs_mutex_unlock(&dpdk_mutex);
>> > + unixctl_command_reply_error(conn, response);
>> > +}
>> > +
>> > /*
>> > * Set virtqueue flags so that we do not receive interrupts.
>> > */
>> > @@ -2404,6 +2481,15 @@ dpdk_common_init(void)
>> > "[netdev] up|down", 1, 2,
>> > netdev_dpdk_set_admin_state, NULL);
>> >
>> > + unixctl_command_register("netdev-dpdk/port-attach",
>> > + "pci address of device", 1, 1,
>> > + netdev_dpdk_port_attach, NULL);
>> > +
>> > + unixctl_command_register("netdev-dpdk/port-detach",
>> > + "port", 1, 1,
>> > + netdev_dpdk_port_detach, NULL);
>> > +
>> > + ovs_thread_create("dpdk_watchdog", dpdk_watchdog, NULL);
>> > }
>> >
>> > /* Client Rings */
>> > @@ -2414,7 +2500,7 @@ dpdk_ring_create(const char dev_name[], unsigned
>> int port_no,
>> > {
>> > struct dpdk_ring *ivshmem;
>> > char ring_name[RTE_RING_NAMESIZE];
>> > - int err;
>> > + int err, port_id;
>> >
>> > ivshmem = dpdk_rte_mzalloc(sizeof *ivshmem);
>> > if (ivshmem == NULL) {
>> > @@ -2448,19 +2534,20 @@ dpdk_ring_create(const char dev_name[],
>> unsigned int port_no,
>> > return ENOMEM;
>> > }
>> >
>> > - err = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1,
>> > - &ivshmem->cring_tx, 1, SOCKET0);
>> > + port_id = rte_eth_from_rings(dev_name, &ivshmem->cring_rx, 1,
>> > + &ivshmem->cring_tx, 1, SOCKET0);
>> >
>> > - if (err < 0) {
>> > + if (port_id < 0) {
>> > rte_free(ivshmem);
>> > return ENODEV;
>> > }
>> >
>> > ivshmem->user_port_id = port_no;
>> > - ivshmem->eth_port_id = rte_eth_dev_count() - 1;
>> > + ivshmem->eth_port_id = port_id;
>> > + *eth_port_id = port_id;
>> > +
>> > ovs_list_push_back(&dpdk_ring_list, &ivshmem->list_node);
>> >
>> > - *eth_port_id = ivshmem->eth_port_id;
>> > return 0;
>> > }
>> >
>> > --
>> > 1.9.1
>> >
>> > _______________________________________________
>> > dev mailing list
>> > dev at openvswitch.org
>> > http://openvswitch.org/mailman/listinfo/dev
>>
>> --
>> fbl
>>
>>
>
More information about the dev
mailing list