[ovs-dev] [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports

Bala Sankaran bsankara at redhat.com
Mon Sep 10 15:47:24 UTC 2018


Hello Tiago,

Here's an incremental diff of the patch 6 which I tested. It looks to pass the tests. I would be submitting the 
new version (v5) applying these changes:

diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
index baa74da..a9247f8 100644
--- a/tests/system-dpdk.at
+++ b/tests/system-dpdk.at
@@ -88,6 +88,10 @@ OVS_DPDK_PRE_CHECK()
 AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
 OVS_DPDK_START()
 
+dnl Find number of sockets
+AT_CHECK([lscpu], [], [stdout])
+AT_CHECK([cat stdout | grep "Socket(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > SOCKET_MEM])
+
 dnl Add userspace bridge and attach it to OVS
 AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
 AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
@@ -111,7 +115,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
 
 dnl Execute testpmd in background
 on_exit "pkill -f -x -9 'tail -f /dev/null'"
-tail -f /dev/null | testpmd --socket-mem=512 \
+tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
            --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
            --vdev="net_tap0,iface=tap0" --file-prefix page0 \
            --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
@@ -183,7 +187,7 @@ ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
 
 dnl Execute testpmd in background
 on_exit "pkill -f -x -9 'tail -f /dev/null'"
-tail -f /dev/null | testpmd --socket-mem=512 \
+tail -f /dev/null | testpmd --socket-mem="$(cat SOCKET_MEM)" --no-pci\
            --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
            --vdev="net_tap0,iface=tap0" --file-prefix page0 \
            --single-file-segments -- -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &


Before I do so, I had a question for you.

Do you suggest that we have the socket-mem option at all? Because it could vary for each system, and also, the latest 
versions of DPDK would dynamically allocate the socket-memory, but earlier versions might have had the socket-memory
option hard-coded - which could in turn be a different value in each system that the tests run on.

Thanks.
Bala.

----- Original Message -----
> From: "Bala Sankaran" <bsankara at redhat.com>
> To: "Tiago Lam" <tiago.lam at intel.com>
> Cc: dev at openvswitch.org, "Aaron Conole" <aconole at redhat.com>, "Ian Stokes" <ian.stokes at intel.com>, "Ciara Loftus"
> <ciara.loftus at intel.com>
> Sent: Monday, 10 September, 2018 9:22:13 AM
> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports
> 
> ----- Original Message -----
> > From: "Tiago Lam" <tiago.lam at intel.com>
> > To: "Bala Sankaran" <bsankara at redhat.com>
> > Cc: dev at openvswitch.org, "Aaron Conole" <aconole at redhat.com>, "Ian Stokes"
> > <ian.stokes at intel.com>, "Ciara Loftus"
> > <ciara.loftus at intel.com>
> > Sent: Wednesday, 5 September, 2018 3:24:46 AM
> > Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via
> > dpdkvhostuser ports
> > 
> > 
> > 
> > On 04/09/2018 18:46, Bala Sankaran wrote:
> > > ----- Original Message -----
> > >> From: "Tiago Lam" <tiago.lam at intel.com>
> > >> To: "Bala Sankaran" <bsankara at redhat.com>, dev at openvswitch.org
> > >> Cc: "Aaron Conole" <aconole at redhat.com>, "Ian Stokes"
> > >> <ian.stokes at intel.com>, "Ciara Loftus" <ciara.loftus at intel.com>
> > >> Sent: Tuesday, 4 September, 2018 6:32:39 AM
> > >> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via
> > >> dpdkvhostuser ports
> > >>
> > >> On 31/08/2018 16:55, Bala Sankaran wrote:
> > >>> This adds a few unit tests to the 'check-dpdk' subsystem that will
> > >>> exercise allocations of two network namespaces, PMDs, and the
> > >>> vhost-user and vhost-user-client code path(separate tests).
> > >>> As part of the tests, userspace bridge is added and attached to OVS.
> > >>> Also, the tap devices created are added to the network namespaces.
> > >>>
> > >>> The ultimatum is to connect the two network namespaces by pinging them.
> > >>>
> > >>> Signed-off-by: Bala Sankaran <bsankara at redhat.com>
> > >>> Co-authored-by: Aaron Conole <aconole at redhat.com>
> > >>> Signed-off-by: Aaron Conole <aconole at redhat.com>
> > >>> ---
> > >>
> > >> Hi Bala,
> > >>
> > >> Thanks for the new version and including the vhostuserclient test - that
> > >> test runs OK on my end, and passes successfully (just a note below).
> > >>
> > >> As I mentioned on v3, I've faced some issues getting the vhostuser test
> > >> to run on my system. I've tried testpmd binaries compiled against DPDK
> > >> 18.02, 18.05, 18.08 and 18.11-rc0. Only 18.02 seemed to work.
> > >>
> > >> After spending some time bisecting this, it comes down to the
> > >> "--socket-mem" option, and providing some memory for all available
> > >> sockets in the system [1]. If one provides "--socket-mem=512" (like
> > >> you're doing on the tests below), pre DPDK 18.05, testpmd would fail
> > >> with an error mentioning it can't allocate memory for a socket 1 (if
> > >> your system has 2 sockets). Post DPDK 18.05, it goes on and doesn't
> > >> complain, but in my case it also fails to forward traffic.
> > >>
> > >> So, this begs the question, does the first test pass for you as is,
> > >> without a second argument to "--socket-mem"? How many sockets does your
> > >> system have?
> > > 
> > > Hello Tiago,
> > > 
> > > I've got 4 sockets in the VM that I tested on. I have been using the
> > > --socket-mem=512 as an argument for testpmd each time I test, I don't
> > > reckon that my tests would work without the argument.
> Hi Tiago,
> 
> > 
> > Is that what `lscpu` reports under "NUMA nodes(s)", if executed inside
> > the VM?
> > 
> > My argument is not around removing "--socket-mem=", but assigning a
> > value to each available socket. If you look into the OVS_DPDK_START
> > macro, in tests/system-dpdk-macros.at, it is doing exactly that by using
> > `lscpu` and grep'ing for "NUMA node(s)". Otherwise I think this test
> > will fail on several systems and pass on some others.
> 
> Apologies for the delayed response. I understand this, I will add this up.
> 
> > 
> > Also, I think you've missed my other comment below, about using "--no-pci".
> > 
> > Tiago.
> > 
> > > 
> > > 
> > > Best,
> > > Bala.
> > > 
> > >>
> > >> [1] Note that all ports are allocated, in my system, in socket 0. But
> > >> still, I have to allocate as little as "1" to get traffic through.
> > >>
> > >>>  tests/system-dpdk.at | 152
> > >>>  ++++++++++++++++++++++++++++++++++++++++++++++++++-
> > >>>  1 file changed, 151 insertions(+), 1 deletion(-)
> > >>>
> > >>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> > >>> index 58dc8aa..baa74da 100644
> > >>> --- a/tests/system-dpdk.at
> > >>> +++ b/tests/system-dpdk.at
> > >>> @@ -1,7 +1,10 @@
> > >>> +m4_define([CONFIGURE_VETH_OFFLOADS],
> > >>> +   [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
> > >>> +
> > >>>  AT_BANNER([OVS-DPDK unit tests])
> > >>>  
> > >>>  dnl
> > >>>  --------------------------------------------------------------------------
> > >>> -dnl Check if EAL init is successfull
> > >>> +dnl Check if EAL init is successful
> > >>>  AT_SETUP([OVS-DPDK datapath - EAL init])
> > >>>  AT_KEYWORDS([dpdk])
> > >>>  OVS_DPDK_PRE_CHECK()
> > >>> @@ -74,3 +77,150 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open
> > >>> vSwitch
> > >>> kernel module is probably
> > >>>  \@EAL: No free hugepages reported in hugepages-1048576kB at d"])
> > >>>  AT_CLEANUP
> > >>>  dnl
> > >>>  --------------------------------------------------------------------------
> > >>> +
> > >>> +
> > >>> +
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>> +dnl Ping vhost-user port
> > >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user ports])
> > >>> +AT_KEYWORDS([dpdk])
> > >>> +OVS_DPDK_PRE_CHECK()
> > >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> > >>> +OVS_DPDK_START()
> > >>> +
> > >>> +dnl Add userspace bridge and attach it to OVS
> > >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> > >>> datapath_type=netdev])
> > >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface
> > >>> dpdkvhostuser0 \
> > >>> +          type=dpdkvhostuser], [],
> > >>> +         [stdout], [stderr])
> > >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> > >>> +
> > >>> +dnl Parse log file
> > >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user server: socket created" \
> > >>> +          ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "Socket $OVS_RUNDIR/dpdkvhostuser0 created for
> > >>> vhost-user
> > >>> port dpdkvhostuser0" \
> > >>> +          ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "VHOST_CONFIG: bind to $OVS_RUNDIR/dpdkvhostuser0"
> > >>> ovs-vswitchd.log], [],
> > >>> +         [stdout])
> > >>> +
> > >>> +dnl Set up namespaces
> > >>> +ADD_NAMESPACES(ns1, ns2)
> > >>> +
> > >>> +dnl Add veth device
> > >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> > >>> +
> > >>> +dnl Execute testpmd in background
> > >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> > >>> +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> > >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> > >>> +           --single-file-segments -- -a
> > >>>> $OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> > >>
> > >> Since this test doesn't require any physical devices, I would use either
> > >> the "--no-pci" option or check if any physical devices is bound to DPDK
> > >> any skip the test if so. Otherwise DPDK will try to init the driver
> > >> there and testpmd will hang as a consequence. Same for the other test
> > >> below.
> 
> I will add the --no-pci option as well to my tests and submit a v5.
> 
> Thank you.
> 
> Best,
> Bala.
> 
> > >>
> > >> Tiago.
> > >>
> > >>> +
> > >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> > >>> form.
> > >>> +sleep 10
> > >>> +
> > >>> +dnl Move the tap devices to the namespaces
> > >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0],
> > >>> [],
> > >>> +         [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> > >>> [stdout],
> > >>> +         [stderr])
> > >>> +
> > >>> +dnl Clean up the testpmd now
> > >>> +pkill -f -x -9 'tail -f /dev/null'
> > >>> +
> > >>> +dnl Clean up
> > >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout],
> > >>> [stderr])
> > >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module
> > >>> is
> > >>> probably not loaded. at d
> > >>> +\@Failed to enable flow control at d
> > >>> +\@VHOST_CONFIG: recvmsg failed at d
> > >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostuser0: No
> > >>> such
> > >>> file or directory at d
> > >>> +\@Global register is changed during at d
> > >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> > >>> dpdkvhostuserclient ports. at d
> > >>> +\@failed to enumerate system datapaths: No such file or directory at d
> > >>> +\@EAL:   Invalid NUMA socket, default to 0 at d
> > >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > >>> unreliable clock cycles !@d
> > >>> +\@EAL: No free hugepages reported in hugepages-1048576kB at d"])
> > >>> +AT_CLEANUP
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>> +
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>> +dnl Ping vhost-user-client port
> > >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user-client ports])
> > >>> +AT_KEYWORDS([dpdk])
> > >>> +OVS_DPDK_PRE_CHECK()
> > >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> > >>> +OVS_DPDK_START()
> > >>> +
> > >>> +dnl Add userspace bridge and attach it to OVS
> > >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> > >>> datapath_type=netdev])
> > >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set
> > >>> Interface
> > >>> \
> > >>> +          dpdkvhostuserclient0 \
> > >>> +          type=dpdkvhostuserclient \
> > >>> +          options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> > >>> +         [stdout], [stderr])
> > >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> > >>> +
> > >>> +dnl Parse log file
> > >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created"
> > >>> ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in
> > >>> 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> > >>> +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0:
> > >>> reconnecting..." ovs-vswitchd.log], [], [stdout])
> > >>> +
> > >>> +dnl Set up namespaces
> > >>> +ADD_NAMESPACES(ns1, ns2)
> > >>> +
> > >>> +dnl Add veth device
> > >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> > >>> +
> > >>> +dnl Execute testpmd in background
> > >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> > >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> > >>> +
> > >>> --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1"
> > >>> \
> > >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> > >>> +           --single-file-segments -- -a
> > >>>> $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> > >>> +
> > >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> > >>> form.
> > >>> +sleep 10
> > >>> +
> > >>> +dnl Move the tap devices to the namespaces
> > >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> > >>> [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0],
> > >>> [],
> > >>> +         [stdout], [stderr])
> > >>> +
> > >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> > >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> > >>> [stdout],
> > >>> +         [stderr])
> > >>> +
> > >>> +dnl Clean up the testpmd now
> > >>> +pkill -f -x -9 'tail -f /dev/null'
> > >>> +
> > >>> +dnl Clean up
> > >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout],
> > >>> [stderr])
> > >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module
> > >>> is
> > >>> probably not loaded. at d
> > >>> +\@Failed to enable flow control at d
> > >>> +\@VHOST_CONFIG: recvmsg failed at d
> > >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No
> > >>> such
> > >>> file or directory at d
> > >>> +\@Global register is changed during at d
> > >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> > >>> dpdkvhostuserclient ports. at d
> > >>> +\@failed to enumerate system datapaths: No such file or directory at d
> > >>> +\@EAL:   Invalid NUMA socket, default to 0 at d
> > >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> > >>> unreliable clock cycles !@d
> > >>> +\@EAL: No free hugepages reported in hugepages-1048576kB at d"])
> > >>> +AT_CLEANUP
> > >>> +dnl
> > >>> --------------------------------------------------------------------------
> > >>>
> > >>
> >


More information about the dev mailing list