[ovs-dev] [PATCH v4] tests: Add system-dpdk-testsuite

Stokes, Ian ian.stokes at intel.com
Wed Apr 18 15:44:15 UTC 2018


> Hi Ian,
> 
> Appreciate your comments and findings. See my replies inline.
> 
> 
> > -----Original Message-----
> > From: Stokes, Ian
> > Sent: Tuesday, April 17, 2018 8:50 PM
> > To: Rybka, MarcinX <marcinx.rybka at intel.com>; dev at openvswitch.org
> > Cc: Rybka, MarcinX <marcinx.rybka at intel.com>
> > Subject: RE: [ovs-dev] [PATCH v4] tests: Add system-dpdk-testsuite
> >
> > > To: dev at openvswitch.org
> > > Cc: Rybka, MarcinX <marcinx.rybka at intel.com>
> > > Subject: [ovs-dev] [PATCH v4] tests: Add system-dpdk-testsuite
> > >
> > > New OVS-DPDK testsuite, which can be launched via `make check-dpdk`,
> > > tests OVS using a DPDK datapath. The testsuite contains already
> > > initial
> > tests:
> > >  1. EAL init
> > >  2. Add standard DPDK PHY port
> > >  3. Add vhost-user-client port
> > >
> > > Signed-off-by: Marcin Rybka <marcinx.rybka at intel.com>
> > >
> >
> > Thanks for this Marcin, comments inline below.
> > > ---
> > > Ver.4 updates:
> > >  - now works with DPDK 17.11.1
> > >  - added entry in NEWS
> > >  - better hugepages allocation
> > > ---
> > >  Documentation/topics/testing.rst | 19 ++++++++++++
> > >  NEWS                             |  2 ++
> > >  tests/automake.mk                | 17 ++++++++++
> > >  tests/system-dpdk-macros.at      | 56
> > +++++++++++++++++++++++++++++++++
> > >  tests/system-dpdk-testsuite.at   | 25 +++++++++++++++
> > >  tests/system-dpdk.at             | 67
> > > ++++++++++++++++++++++++++++++++++++++++
> > >  6 files changed, 186 insertions(+)
> > >  create mode 100644 tests/system-dpdk-macros.at  create mode 100644
> > > tests/system-dpdk-testsuite.at  create mode 100644
> > > tests/system-dpdk.at
> > >
> > > diff --git a/Documentation/topics/testing.rst
> > > b/Documentation/topics/testing.rst
> > > index 5dcf446..bdc7225 100644
> > > --- a/Documentation/topics/testing.rst
> > > +++ b/Documentation/topics/testing.rst
> > > @@ -297,6 +297,25 @@ To invoke the datapath testsuite with the
> > > userspace datapath, run::
> > >
> > >  The results of the testsuite are in ``tests/system-userspace-
> > > testsuite.dir``.
> > >
> > > +DPDK datapath
> > > +'''''''''''''
> > > +
> > > +To test :doc:`/intro/install/dpdk` (i.e., the build was configured
> > > +with ``--with-dpdk``, the DPDK is installed), run the testsuite and
> > > +generate a report by using the ``check-dpdk`` target::
> > > +
> > > +    $ make check-dpdk
> > > +
> > > +To see a list of all the available tests, run::
> > > +
> > > +    $ make check-dpdk TESTSUITEFLAGS=--list
> > > +
> > > +These tests require a `DPDK supported NIC`_ and proper DPDK
> > > +variables (``DPDK_DIR`` and ``DPDK_BUILD``). Moreover you need to
> > > +have root privileges, load the required modules and bind the NIC to
> > > +the DPDK-
> > > compatible driver.
> > > +
> > > +.. _DPDK supported NIC: http://dpdk.org/doc/nics
> > > +
> > >  Kernel datapath
> > >  '''''''''''''''
> > >
> > > diff --git a/NEWS b/NEWS
> > > index 58a7b58..7bad608 100644
> > > --- a/NEWS
> > > +++ b/NEWS
> > > @@ -21,6 +21,8 @@ Post-v2.9.0
> > >         and reply with a RST for TCP or ICMPv4/ICMPv6 unreachable
> > > message for
> > >         other IPv4/IPv6-based protocols whenever a reject ACL rule is
> hit.
> > >       * ACL match conditions can now match on Port_Groups.
> > > +   - New 'check-dpdk' Makefile target to run a new system testsuite.
> > > +     See Testing topic for the details.
> >
> > I think this should be added under a specific DPDK header in NEWS.
> >
> > - DPDK
> >   * Add 'check-dpdk' Makefile target to run dpdk specific testsuite.
> >
> > Thoughts?
> 
> Yes, I can do like this. I was suggested by previous news about new
> testsuites.

Sure, I guess in the past unit tests have been missing for DPDK specifically so it's nice to call it out under that header.

> 
> >
> > >
> > >  v2.9.0 - 19 Feb 2018
> > >  --------------------
> > > diff --git a/tests/automake.mk b/tests/automake.mk index
> > > d9292e8..e52531a
> > > 100644
> > > --- a/tests/automake.mk
> > > +++ b/tests/automake.mk
> > > @@ -5,10 +5,12 @@ EXTRA_DIST += \
> > >  	$(SYSTEM_KMOD_TESTSUITE_AT) \
> > >  	$(SYSTEM_USERSPACE_TESTSUITE_AT) \
> > >  	$(SYSTEM_OFFLOADS_TESTSUITE_AT) \
> > > +	$(SYSTEM_DPDK_TESTSUITE_AT) \
> > >  	$(TESTSUITE) \
> > >  	$(SYSTEM_KMOD_TESTSUITE) \
> > >  	$(SYSTEM_USERSPACE_TESTSUITE) \
> > >  	$(SYSTEM_OFFLOADS_TESTSUITE) \
> > > +	$(SYSTEM_DPDK_TESTSUITE) \
> > >  	tests/atlocal.in \
> > >  	$(srcdir)/package.m4 \
> > >  	$(srcdir)/tests/testsuite \
> > > @@ -128,6 +130,12 @@ SYSTEM_OFFLOADS_TESTSUITE_AT = \
> > >  	tests/system-offloads-traffic.at \
> > >  	tests/system-offloads-testsuite.at
> > >
> > > +SYSTEM_DPDK_TESTSUITE_AT = \
> > > +	tests/system-common-macros.at \
> > > +	tests/system-dpdk-macros.at \
> > > +	tests/system-dpdk-testsuite.at \
> > > +	tests/system-dpdk.at
> > > +
> > >  check_SCRIPTS += tests/atlocal
> > >
> > >  TESTSUITE = $(srcdir)/tests/testsuite @@ -135,6 +143,7 @@
> > > TESTSUITE_PATCH = $(srcdir)/tests/testsuite.patch
> > > SYSTEM_KMOD_TESTSUITE = $(srcdir)/tests/system-kmod-testsuite
> > >  SYSTEM_USERSPACE_TESTSUITE =
> > > $(srcdir)/tests/system-userspace-testsuite
> > >  SYSTEM_OFFLOADS_TESTSUITE =
> > > $(srcdir)/tests/system-offloads-testsuite
> > > +SYSTEM_DPDK_TESTSUITE = $(srcdir)/tests/system-dpdk-testsuite
> > >  DISTCLEANFILES += tests/atconfig tests/atlocal
> > >
> > >  AUTOTEST_PATH =
> > > utilities:vswitchd:ovsdb:vtep:tests:$(PTHREAD_WIN32_DIR_DLL):$(SSL_D
> > > IR ):ov n/controller-vtep:ovn/northd:ovn/utilities:ovn/controller
> > > @@ -258,6 +267,10 @@ check-offloads: all
> > >  	set $(SHELL) '$(SYSTEM_OFFLOADS_TESTSUITE)' -C tests
> > > AUTOTEST_PATH='$(AUTOTEST_PATH)' $(TESTSUITEFLAGS) -j1; \
> > >  	"$$@" || (test X'$(RECHECK)' = Xyes && "$$@" --recheck)
> > >
> > > +check-dpdk: all
> > > +	set $(SHELL) '$(SYSTEM_DPDK_TESTSUITE)' -C tests
> > > AUTOTEST_PATH='$(AUTOTEST_PATH)' $(TESTSUITEFLAGS) -j1; \
> > > +	"$$@" || (test X'$(RECHECK)' = Xyes && "$$@" --recheck)
> > > +
> > >  clean-local:
> > >  	test ! -f '$(TESTSUITE)' || $(SHELL) '$(TESTSUITE)' -C tests
> > > --clean
> > >
> > > @@ -286,6 +299,10 @@ $(SYSTEM_OFFLOADS_TESTSUITE): package.m4
> > > $(SYSTEM_TESTSUITE_AT) $(SYSTEM_OFFLOAD
> > >  	$(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
> > >  	$(AM_V_at)mv $@.tmp $@
> > >
> > > +$(SYSTEM_DPDK_TESTSUITE): package.m4 $(SYSTEM_TESTSUITE_AT)
> > > $(SYSTEM_DPDK_TESTSUITE_AT) $(COMMON_MACROS_AT)
> > > +	$(AM_V_GEN)$(AUTOTEST) -I '$(srcdir)' -o $@.tmp $@.at
> > > +	$(AM_V_at)mv $@.tmp $@
> > > +
> > >  # The `:;' works around a Bash 3.2 bug when the output is not
> writeable.
> > >  $(srcdir)/package.m4: $(top_srcdir)/configure.ac
> > >  	$(AM_V_GEN):;{ \
> > > diff --git a/tests/system-dpdk-macros.at
> > > b/tests/system-dpdk-macros.at new file mode 100644 index
> > > 0000000..0762ee0
> > > --- /dev/null
> > > +++ b/tests/system-dpdk-macros.at
> > > @@ -0,0 +1,56 @@
> > > +# OVS_DPDK_PRE_CHECK()
> > > +#
> > > +# Check prerequisites for DPDK tests. Following settings are checked:
> > > +#  - Hugepages
> > > +#  - UIO driver
> > > +#
> > > +m4_define([OVS_DPDK_PRE_CHECK],
> > > +  [dnl Check Hugepages
> > > +   AT_CHECK([cat /proc/meminfo], [], [stdout])
> > > +   AT_CHECK([grep HugePages_ stdout], [], [stdout])
> > > +   AT_CHECK([mount], [], [stdout])
> > > +   AT_CHECK([grep 'hugetlbfs' stdout], [], [stdout], [])
> > > +
> > > +   dnl Check if VFIO or UIO driver is loaded
> > > +   AT_CHECK([lsmod | grep -E "igb_uio|vfio"], [], [stdout])
> > > +
> > > +   dnl Find PCI address candidate, skip if there is no
> > > + DPDK-compatible
> > > NIC
> > > +   AT_CHECK([$DPDK_DIR/usertools/dpdk-devbind.py -s | head -n +4 |
> > > + tail -
> > > 1], [], [stdout])
> >
> > In testing I found I had to explicitly set DPDK_DIR as part of
> > /etc/environment.
> >
> > Defining it as part of .profile or even exporting from the bash shell
> > explicitly before running the tests would result in the DPDK_DIR being
> empty.
> >
> > Not sure why this is. It could be specific to my setup. Have you
> > thought about passing the DPDK_DIR as an optional argument as part of
> > running ./configure rather than relying on environmental variable
> declarations?
> 
> Bad news is that it looks like it's not working...
> Good news is that I finally reproduced your issue (it took me quite a lot
> of time).
> Working on that...

Ok that's good to hear. If there's any testing I can help with let me know.

Ian
> 
> >
> > > +   AT_CHECK([cat stdout | cut -d" " -s -f1 > PCI_ADDR])
> > > +   AT_CHECK([test -s PCI_ADDR || exit 77])
> > > +])
> > > +
> > > +
> > > +# OVS_DPDK_START()
> > > +#
> > > +# Create an empty database and start ovsdb-server. Add special
> > > +configuration # dpdk-init to enable DPDK functionality. Start
> > > +ovs-vswitchd connected to that # database using system devices (no
> > > dummies).
> > > +#
> > > +m4_define([OVS_DPDK_START],
> > > +  [dnl Create database.
> > > +   AT_CHECK([touch .conf.db.~lock~])
> > > +   AT_CHECK([ovsdb-tool create conf.db
> > > +$abs_top_srcdir/vswitchd/vswitch.ovsschema])
> > > +
> > > +   dnl Start ovsdb-server.
> > > +   AT_CHECK([ovsdb-server --detach --no-chdir --pidfile --log-file
> > > + --
> > > remote=punix:$OVS_RUNDIR/db.sock], [0], [stdout], [stderr])
> > > +   on_exit "kill `cat ovsdb-server.pid`"
> > > +   AT_CHECK([[sed < stderr '
> > > +/vlog|INFO|opened log file/d
> > > +/ovsdb_server|INFO|ovsdb-server (Open vSwitch)/d']])
> > > +   AT_CAPTURE_FILE([ovsdb-server.log])
> > > +
> > > +   dnl Initialize database.
> > > +   AT_CHECK([ovs-vsctl --no-wait init])
> > > +
> > > +   dnl Enable DPDK functionality
> > > +   AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch .
> > > + other_config:dpdk-
> > > init=true])
> > > +   AT_CHECK([lscpu], [], [stdout])
> > > +   AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while
> > > (c++<$(3)) {printf "1024,"}; print "1024"}' > SOCKET_MEM])
> > > +   AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch .
> > > + other_config:dpdk-socket-mem="$(cat SOCKET_MEM)"])
> > > +
> > > +   dnl Start ovs-vswitchd.
> > > +   AT_CHECK([ovs-vswitchd --detach --no-chdir --pidfile --log-file
> > > + -
> > > vvconn -vofproto_dpif -vunixctl], [0], [stdout], [stderr])
> > > +   AT_CAPTURE_FILE([ovs-vswitchd.log])
> > > +   on_exit "kill_ovs_vswitchd `cat ovs-vswitchd.pid`"
> > > +])
> > > diff --git a/tests/system-dpdk-testsuite.at b/tests/system-dpdk-
> > > testsuite.at new file mode 100644 index 0000000..382f09e
> > > --- /dev/null
> > > +++ b/tests/system-dpdk-testsuite.at
> > > @@ -0,0 +1,25 @@
> > > +AT_INIT
> > > +
> > > +AT_COPYRIGHT([Copyright (c) 2017 Intel Corporation
> > > +
> > > +Licensed under the Apache License, Version 2.0 (the "License"); you
> > > +may not use this file except in compliance with the License.
> > > +You may obtain a copy of the License at:
> > > +
> > > +    http://www.apache.org/licenses/LICENSE-2.0
> > > +
> > > +Unless required by applicable law or agreed to in writing, software
> > > +distributed under the License is distributed on an "AS IS" BASIS,
> > > +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > implied.
> > > +See the License for the specific language governing permissions and
> > > +limitations under the License.])
> > > +
> > > +m4_ifdef([AT_COLOR_TESTS], [AT_COLOR_TESTS])
> > > +
> > > +m4_include([tests/ovs-macros.at])
> > > +m4_include([tests/ovsdb-macros.at])
> > > +m4_include([tests/ofproto-macros.at])
> > > +m4_include([tests/system-common-macros.at])
> > > +m4_include([tests/system-dpdk-macros.at])
> > > +
> > > +m4_include([tests/system-dpdk.at])
> > > diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at new file
> > > mode
> > > 100644 index 0000000..1f1839e
> > > --- /dev/null
> > > +++ b/tests/system-dpdk.at
> > > @@ -0,0 +1,67 @@
> > > +AT_BANNER([OVS-DPDK unit tests])
> > > +
> > > +dnl
> > > +-------------------------------------------------------------------
> > > +--
> > > +--
> > > +---
> > > +dnl Check if EAL init is successfull AT_SETUP([OVS-DPDK datapath -
> > > +EAL init])
> > > +AT_KEYWORDS([dpdk])
> > > +dnl OVS_DPDK_PRE_CHECK()
> > > +OVS_DPDK_START()
> > > +AT_CHECK([grep "DPDK Enabled - initializing..." ovs-vswitchd.log],
> > > +[],
> > > +[stdout]) AT_CHECK([grep "EAL" ovs-vswitchd.log], [], [stdout])
> > > +AT_CHECK([grep "DPDK Enabled - initialized" ovs-vswitchd.log], [],
> > > +[stdout]) OVS_VSWITCHD_STOP("/Global register is changed during/d")
> > > +AT_CLEANUP dnl
> > > +-------------------------------------------------------------------
> > > +--
> > > +--
> > > +---
> > > +
> > > +
> > > +
> > > +dnl
> > > +-------------------------------------------------------------------
> > > +--
> > > +--
> > > +---
> > > +dnl Add standard DPDK PHY port
> > > +AT_SETUP([OVS-DPDK datapath - add standard DPDK port])
> > > +AT_KEYWORDS([dpdk])
> > > +
> > > +OVS_DPDK_PRE_CHECK()
> > > +OVS_DPDK_START()
> > > +
> > > +dnl Add userspace bridge and attach it to OVS AT_CHECK([ovs-vsctl
> > > +add-br br10 -- set bridge br10 datapath_type=netdev])
> > > +AT_CHECK([ovs-vsctl add-port br10 phy0 -- set Interface phy0
> > > +type=dpdk options:dpdk-devargs=$(cat PCI_ADDR)], [], [stdout],
> > > +[stderr]) AT_CHECK([ovs-vsctl show], [], [stdout]) sleep 2
> > > +
> > > +dnl Clean up
> > > +AT_CHECK([ovs-vsctl del-port br10 phy0], [], [stdout], [stderr])
> > > +OVS_VSWITCHD_STOP("/does not exist. The Open vSwitch kernel
> > module is
> > > +probably not loaded./d /Failed to enable flow control/d /Global
> > > +register is changed during/d") AT_CLEANUP dnl
> > > +-------------------------------------------------------------------
> > > +--
> > > +--
> > > +---
> > > +
> > > +
> > > +
> > > +dnl
> > > +-------------------------------------------------------------------
> > > +--
> > > +--
> > > +---
> > > +dnl Add vhost-user-client port
> > > +AT_SETUP([OVS-DPDK datapath - add vhost-user-client port])
> > > +AT_KEYWORDS([dpdk])
> > > +
> > > +OVS_DPDK_START()
> > > +
> > > +dnl Add userspace bridge and attach it to OVS AT_CHECK([ovs-vsctl
> > > +add-br br10 -- set bridge br10 datapath_type=netdev])
> > > +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set
> > > +Interface
> > > +dpdkvhostuserclient0 type=dpdkvhostuserclient
> > > +options:vhost-server-path=/tmp/dpdkvhostclient0], [], [stdout],
> > > +[stderr]) AT_CHECK([ovs-vsctl show], [], [stdout]) sleep 2
> > > +
> > > +dnl Parse log file
> > > +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created"
> > > +ovs-vswitchd.log], [], [stdout]) AT_CHECK([grep "vHost User device
> > > +'dpdkvhostuserclient0' created in 'client' mode, using client socket"
> > > +ovs-vswitchd.log], [], [stdout]) AT_CHECK([grep "VHOST_CONFIG:
> > > +/tmp/dpdkvhostclient0: reconnecting..." ovs-vswitchd.log], [],
> > > +[stdout])
> > > +
> > > +dnl Clean up
> > > +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [],
> > > +[stdout],
> > > +[stderr]) OVS_VSWITCHD_STOP("/does not exist. The Open vSwitch
> > kernel
> > > +module is probably not loaded./d /Failed to enable flow control/d
> > > +/failed to connect to \/tmp\/dpdkvhostclient0: No such file or
> > > +directory/d /Global register is changed during/d") AT_CLEANUP dnl
> > > +-------------------------------------------------------------------
> > > +--
> > > +--
> > > +---
> > > --
> > > 1.9.3
> > >
> > > _______________________________________________
> > > dev mailing list
> > > dev at openvswitch.org
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev


More information about the dev mailing list