[ovs-dev] [PATCH V6 1/2] netdev-dpdk: DPDK v17.11 upgrade

Fri Dec 8 13:22:26 UTC 2017

Sorry for the late comments.
I'm still not happy with the proposed documentation changes (see below).

/Jan

> -----Original Message-----
> From: Mark Kavanagh [mailto:mark.b.kavanagh at intel.com]
> Sent: Friday, 08 December, 2017 11:54
> To: dev at openvswitch.org
> Cc: ktraynor at redhat.com; maxime.coquelin at redhat.com; i.maximets at samsung.com; Jan Scheurich <jan.scheurich at ericsson.com>;
> sean.k.mooney at intel.com; ian.stokes at intel.com; ligs at dtdream.com; antonio.fischetti at intel.com; tiwei.bie at intel.com;
> ciara.loftus at intel.com; Mark Kavanagh <mark.b.kavanagh at intel.com>
> Subject: [ovs-dev][PATCH V6 1/2] netdev-dpdk: DPDK v17.11 upgrade
> 
> This commit adds support for DPDK v17.11:
> - minor updates to accomodate DPDK API changes
> - update references to DPDK version in Documentation
> - update DPDK version in travis' linux-build script
> - document DPDK v17.11 virtio driver bug
> 
> Signed-off-by: Mark Kavanagh <mark.b.kavanagh at intel.com>
> Acked-by: Maxime Coquelin <maxime.coquelin at redhat.com>
> Acked-by: Ciara Loftus <ciara.loftus at intel.com>
> Acked-by: Jan Scheurich <jan.scheurich at ericsson.com>
> Tested-by: Jan Scheurich <jan.scheurich at ericsson.com>
> Tested-by: Guoshuai Li <ligs at dtdream.com>
> ---
>  .travis/linux-build.sh                   |  2 +-
>  Documentation/faq/releases.rst           |  1 +
>  Documentation/intro/install/dpdk.rst     | 10 +++++-----
>  Documentation/topics/dpdk/ring.rst       |  2 +-
>  Documentation/topics/dpdk/vhost-user.rst | 32 +++++++++++++++++++++++++++-----
>  NEWS                                     |  2 ++
>  lib/netdev-dpdk.c                        |  5 +++--
>  7 files changed, 40 insertions(+), 14 deletions(-)
> 
> diff --git a/.travis/linux-build.sh b/.travis/linux-build.sh
> index 4d6459f..ed28ee4 100755
> --- a/.travis/linux-build.sh
> +++ b/.travis/linux-build.sh
> @@ -81,7 +81,7 @@ fi
> 
>  if [ "$DPDK" ]; then
>      if [ -z "$DPDK_VER" ]; then
> -        DPDK_VER="17.05.2"
> +        DPDK_VER="17.11"
>      fi
>      install_dpdk $DPDK_VER
>      if [ "$CC" = "clang" ]; then
> diff --git a/Documentation/faq/releases.rst b/Documentation/faq/releases.rst
> index d903b06..62a1957 100644
> --- a/Documentation/faq/releases.rst
> +++ b/Documentation/faq/releases.rst
> @@ -164,6 +164,7 @@ Q: What DPDK version does each Open vSwitch release work with?
>      2.6.x        16.07.2
>      2.7.x        16.11.3
>      2.8.x        17.05.2
> +    2.9.x        17.11
>      ============ =======
> 
>  Q: I get an error like this when I configure Open vSwitch:
> diff --git a/Documentation/intro/install/dpdk.rst b/Documentation/intro/install/dpdk.rst
> index bb69ae5..3fecb5c 100644
> --- a/Documentation/intro/install/dpdk.rst
> +++ b/Documentation/intro/install/dpdk.rst
> @@ -40,7 +40,7 @@ Build requirements
>  In addition to the requirements described in :doc:`general`, building Open
>  vSwitch with DPDK will require the following:
> 
> -- DPDK 17.05.2
> +- DPDK 17.11
> 
>  - A `DPDK supported NIC`_
> 
> @@ -69,9 +69,9 @@ Install DPDK
>  #. Download the `DPDK sources`_, extract the file and set ``DPDK_DIR``::
> 
>         $ cd /usr/src/
> -       $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz
> -       $ tar xf dpdk-17.05.2.tar.xz
> -       $ export DPDK_DIR=/usr/src/dpdk-stable-17.05.2
> +       $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz
> +       $ tar xf dpdk-17.11.tar.xz
> +       $ export DPDK_DIR=/usr/src/dpdk-17.11
>         $ cd $DPDK_DIR
> 
>  #. (Optional) Configure DPDK as a shared library
> @@ -583,7 +583,7 @@ Limitations
>    The latest list of validated firmware versions can be found in the `DPDK
>    release notes`_.
> 
> -.. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_17_05.html
> +.. _DPDK release notes: http://dpdk.org/doc/guides/rel_notes/release_17_11.html
> 

What happened to my request to add here that running OVS 2.9 with DPDK datapath in a VM with virtio interfaces may itself suffer from the DPDK 17.11 bug in OVS' own virtio PMD? The work-around for this could be to enable the mrg_rxbuf feature in Qemu for the OVS VM as that would trigger OVS to not use vector rx. 

This is essentially different from the note in the vhost_user section below that talks about DPDK apps running in a guest VM served by OVS.

>  Reporting Bugs
>  --------------
> diff --git a/Documentation/topics/dpdk/ring.rst b/Documentation/topics/dpdk/ring.rst
> index ad9d7a5..8d0ede8 100644
> --- a/Documentation/topics/dpdk/ring.rst
> +++ b/Documentation/topics/dpdk/ring.rst
> @@ -77,4 +77,4 @@ DPDK. However, this functionality was removed because:
>  - :doc:`vhost-user interfaces <vhost-user>` are the defacto DPDK-based path to
>    guests
> 
> -.. _DPDK documentation: https://dpdk.readthedocs.io/en/v17.05/prog_guide/ring_lib.html
> +.. _DPDK documentation: https://dpdk.readthedocs.io/en/v17.11/prog_guide/ring_lib.html
> diff --git a/Documentation/topics/dpdk/vhost-user.rst b/Documentation/topics/dpdk/vhost-user.rst
> index 74ac06e..a43affa 100644
> --- a/Documentation/topics/dpdk/vhost-user.rst
> +++ b/Documentation/topics/dpdk/vhost-user.rst
> @@ -292,9 +292,9 @@ To begin, instantiate a guest as described in :ref:`dpdk-vhost-user` or
>  DPDK sources to VM and build DPDK::
> 
>      $ cd /root/dpdk/
> -    $ wget http://fast.dpdk.org/rel/dpdk-17.05.2.tar.xz
> -    $ tar xf dpdk-17.05.2.tar.xz
> -    $ export DPDK_DIR=/root/dpdk/dpdk-stable-17.05.2
> +    $ wget http://fast.dpdk.org/rel/dpdk-17.11.tar.xz
> +    $ tar xf dpdk-17.11.tar.xz
> +    $ export DPDK_DIR=/root/dpdk/dpdk-17.11
>      $ export DPDK_TARGET=x86_64-native-linuxapp-gcc
>      $ export DPDK_BUILD=$DPDK_DIR/$DPDK_TARGET
>      $ cd $DPDK_DIR
> @@ -327,6 +327,28 @@ Finally, start the application::
> 
>      # TODO
> 
> +.. important::
> +
> +  DPDK v17.11 virtio PMD contains a bug in the vectorized Rx function that
> +  affects testpmd/DPDK guest applications. As such, guest DPDK applications
> +  should use a non-vectorized Rx function.
> +
> +The DPDK v17.11 virtio net driver contains a bug that prevents guest DPDK
> +applications from receiving packets when the vectorized Rx function is used.
> +This only occurs when guest-bound traffic is live before a DPDK application is
> +started within the guest, and where two or more forwarding cores are used. As
> +such, it is not recommended for guests which execute DPDK applications to use
> +the virtio vectorized Rx function. A simple method of ensuring that a non-
> +vectorized Rx function is used is to enable mergeable buffers for the guest,
> +with the following QEMU command line option::
> +
> +    mrg_rxbuf=on

For an arbitrary DPDK application in the guest that may not solve the issue! Where or not mergeable Rx buffers are negotiated to be used depends on two things: 1. Support enabled in Qemu, 2. Support signaled in the guest virtio driver and 3. Support signaled in the vhostuser backend. 

If the application e.g. for performance reasons doesn't want to use mergeable Rx buffers, it could still negotiate to off and apply the broken vector rx routine.

> +
> +Additional details regarding the virtio driver bug are available on the
> +`DPDK mailing list`_.
> +
> +.. _DPDK mailing list: http://dpdk.org/ml/archives/dev/2017-December/082801.html
> +
>  .. _dpdk-vhost-user-xml:
> 
>  Sample XML
> @@ -387,7 +409,7 @@ Sample XML
>            <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser0' mode='client'/>
>             <model type='virtio'/>
>            <driver queues='2'>
> -            <host mrg_rxbuf='off'/>
> +            <host mrg_rxbuf='on'/>

Are you changing this example because of the current bug? Or is there some other reason?

I don't think the temporary bug in DPDK 17.11 justifies to change the example VM XML to enable mrg_rxbuf. This costs performance in OVS and DPDK applications should use it only when needed (e.g. because jumbo frames must be received fragmented into smaller mbufs).

>            </driver>
>          </interface>
>          <interface type='vhostuser'>
> @@ -395,7 +417,7 @@ Sample XML
>            <source type='unix' path='/usr/local/var/run/openvswitch/dpdkvhostuser1' mode='client'/>
>            <model type='virtio'/>
>            <driver queues='2'>
> -            <host mrg_rxbuf='off'/>
> +            <host mrg_rxbuf='on'/>
>            </driver>
>          </interface>
>          <serial type='pty'>
> diff --git a/NEWS b/NEWS
> index 427c8f8..d4a1c9a 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -13,6 +13,8 @@ Post-v2.8.0
>       * ovn-ctl: New commands run_nb_ovsdb and run_sb_ovsdb.
>     - Linux kernel 4.13
>       * Add support for compiling OVS with the latest Linux 4.13 kernel
> +   - DPDK:
> +     * Add support for DPDK v17.11
> 
>  v2.8.0 - 31 Aug 2017
>  --------------------
> diff --git a/lib/netdev-dpdk.c b/lib/netdev-dpdk.c
> index faff842..f552444 100644
> --- a/lib/netdev-dpdk.c
> +++ b/lib/netdev-dpdk.c
> @@ -26,6 +26,7 @@
>  #include <sys/socket.h>
>  #include <linux/if.h>
> 
> +#include <rte_bus_pci.h>
>  #include <rte_config.h>
>  #include <rte_cycles.h>
>  #include <rte_errno.h>
> @@ -140,8 +141,8 @@ static struct vlog_rate_limit rl = VLOG_RATE_LIMIT_INIT(5, 20);
> 
>  #define DPDK_ETH_PORT_ID_INVALID    RTE_MAX_ETHPORTS
> 
> -/* DPDK library uses uint8_t for port_id. */
> -typedef uint8_t dpdk_port_t;
> +/* DPDK library uses uint16_t for port_id. */
> +typedef uint16_t dpdk_port_t;
> 
>  #define VHOST_ENQ_RETRY_NUM 8
>  #define IF_NAME_SZ (PATH_MAX > IFNAMSIZ ? PATH_MAX : IFNAMSIZ)
> --
> 1.9.3