[ovs-dev] [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports

Bala Sankaran bsankara at redhat.com
Mon Sep 10 13:22:13 UTC 2018 

----- Original Message -----
> From: "Tiago Lam" <tiago.lam at intel.com>
> To: "Bala Sankaran" <bsankara at redhat.com>
> Cc: dev at openvswitch.org, "Aaron Conole" <aconole at redhat.com>, "Ian Stokes" <ian.stokes at intel.com>, "Ciara Loftus"
> <ciara.loftus at intel.com>
> Sent: Wednesday, 5 September, 2018 3:24:46 AM
> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via dpdkvhostuser ports
> 
> 
> 
> On 04/09/2018 18:46, Bala Sankaran wrote:
> > ----- Original Message -----
> >> From: "Tiago Lam" <tiago.lam at intel.com>
> >> To: "Bala Sankaran" <bsankara at redhat.com>, dev at openvswitch.org
> >> Cc: "Aaron Conole" <aconole at redhat.com>, "Ian Stokes"
> >> <ian.stokes at intel.com>, "Ciara Loftus" <ciara.loftus at intel.com>
> >> Sent: Tuesday, 4 September, 2018 6:32:39 AM
> >> Subject: Re: [PATCH v4 6/6] system-dpdk: Connect network namespaces via
> >> dpdkvhostuser ports
> >>
> >> On 31/08/2018 16:55, Bala Sankaran wrote:
> >>> This adds a few unit tests to the 'check-dpdk' subsystem that will
> >>> exercise allocations of two network namespaces, PMDs, and the
> >>> vhost-user and vhost-user-client code path(separate tests).
> >>> As part of the tests, userspace bridge is added and attached to OVS.
> >>> Also, the tap devices created are added to the network namespaces.
> >>>
> >>> The ultimatum is to connect the two network namespaces by pinging them.
> >>>
> >>> Signed-off-by: Bala Sankaran <bsankara at redhat.com>
> >>> Co-authored-by: Aaron Conole <aconole at redhat.com>
> >>> Signed-off-by: Aaron Conole <aconole at redhat.com>
> >>> ---
> >>
> >> Hi Bala,
> >>
> >> Thanks for the new version and including the vhostuserclient test - that
> >> test runs OK on my end, and passes successfully (just a note below).
> >>
> >> As I mentioned on v3, I've faced some issues getting the vhostuser test
> >> to run on my system. I've tried testpmd binaries compiled against DPDK
> >> 18.02, 18.05, 18.08 and 18.11-rc0. Only 18.02 seemed to work.
> >>
> >> After spending some time bisecting this, it comes down to the
> >> "--socket-mem" option, and providing some memory for all available
> >> sockets in the system [1]. If one provides "--socket-mem=512" (like
> >> you're doing on the tests below), pre DPDK 18.05, testpmd would fail
> >> with an error mentioning it can't allocate memory for a socket 1 (if
> >> your system has 2 sockets). Post DPDK 18.05, it goes on and doesn't
> >> complain, but in my case it also fails to forward traffic.
> >>
> >> So, this begs the question, does the first test pass for you as is,
> >> without a second argument to "--socket-mem"? How many sockets does your
> >> system have?
> > 
> > Hello Tiago,
> > 
> > I've got 4 sockets in the VM that I tested on. I have been using the
> > --socket-mem=512 as an argument for testpmd each time I test, I don't
> > reckon that my tests would work without the argument.
Hi Tiago,

> 
> Is that what `lscpu` reports under "NUMA nodes(s)", if executed inside
> the VM?
> 
> My argument is not around removing "--socket-mem=", but assigning a
> value to each available socket. If you look into the OVS_DPDK_START
> macro, in tests/system-dpdk-macros.at, it is doing exactly that by using
> `lscpu` and grep'ing for "NUMA node(s)". Otherwise I think this test
> will fail on several systems and pass on some others.

Apologies for the delayed response. I understand this, I will add this up.

> 
> Also, I think you've missed my other comment below, about using "--no-pci".
> 
> Tiago.
> 
> > 
> > 
> > Best,
> > Bala.
> > 
> >>
> >> [1] Note that all ports are allocated, in my system, in socket 0. But
> >> still, I have to allocate as little as "1" to get traffic through.
> >>
> >>>  tests/system-dpdk.at | 152
> >>>  ++++++++++++++++++++++++++++++++++++++++++++++++++-
> >>>  1 file changed, 151 insertions(+), 1 deletion(-)
> >>>
> >>> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> >>> index 58dc8aa..baa74da 100644
> >>> --- a/tests/system-dpdk.at
> >>> +++ b/tests/system-dpdk.at
> >>> @@ -1,7 +1,10 @@
> >>> +m4_define([CONFIGURE_VETH_OFFLOADS],
> >>> +   [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
> >>> +
> >>>  AT_BANNER([OVS-DPDK unit tests])
> >>>  
> >>>  dnl
> >>>  --------------------------------------------------------------------------
> >>> -dnl Check if EAL init is successfull
> >>> +dnl Check if EAL init is successful
> >>>  AT_SETUP([OVS-DPDK datapath - EAL init])
> >>>  AT_KEYWORDS([dpdk])
> >>>  OVS_DPDK_PRE_CHECK()
> >>> @@ -74,3 +77,150 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open
> >>> vSwitch
> >>> kernel module is probably
> >>>  \@EAL: No free hugepages reported in hugepages-1048576kB at d"])
> >>>  AT_CLEANUP
> >>>  dnl
> >>>  --------------------------------------------------------------------------
> >>> +
> >>> +
> >>> +
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>> +dnl Ping vhost-user port
> >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user ports])
> >>> +AT_KEYWORDS([dpdk])
> >>> +OVS_DPDK_PRE_CHECK()
> >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> >>> +OVS_DPDK_START()
> >>> +
> >>> +dnl Add userspace bridge and attach it to OVS
> >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> >>> datapath_type=netdev])
> >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface
> >>> dpdkvhostuser0 \
> >>> +          type=dpdkvhostuser], [],
> >>> +         [stdout], [stderr])
> >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> >>> +
> >>> +dnl Parse log file
> >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user server: socket created" \
> >>> +          ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "Socket $OVS_RUNDIR/dpdkvhostuser0 created for vhost-user
> >>> port dpdkvhostuser0" \
> >>> +          ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "VHOST_CONFIG: bind to $OVS_RUNDIR/dpdkvhostuser0"
> >>> ovs-vswitchd.log], [],
> >>> +         [stdout])
> >>> +
> >>> +dnl Set up namespaces
> >>> +ADD_NAMESPACES(ns1, ns2)
> >>> +
> >>> +dnl Add veth device
> >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> >>> +
> >>> +dnl Execute testpmd in background
> >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> >>> +           --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> >>> +           --single-file-segments -- -a
> >>>> $OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> >>
> >> Since this test doesn't require any physical devices, I would use either
> >> the "--no-pci" option or check if any physical devices is bound to DPDK
> >> any skip the test if so. Otherwise DPDK will try to init the driver
> >> there and testpmd will hang as a consequence. Same for the other test
> >> below.

I will add the --no-pci option as well to my tests and submit a v5.

Thank you.

Best,
Bala.

> >>
> >> Tiago.
> >>
> >>> +
> >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> >>> form.
> >>> +sleep 10
> >>> +
> >>> +dnl Move the tap devices to the namespaces
> >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> >>> +         [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> >>> [stdout],
> >>> +         [stderr])
> >>> +
> >>> +dnl Clean up the testpmd now
> >>> +pkill -f -x -9 'tail -f /dev/null'
> >>> +
> >>> +dnl Clean up
> >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout],
> >>> [stderr])
> >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
> >>> probably not loaded. at d
> >>> +\@Failed to enable flow control at d
> >>> +\@VHOST_CONFIG: recvmsg failed at d
> >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostuser0: No such
> >>> file or directory at d
> >>> +\@Global register is changed during at d
> >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> >>> dpdkvhostuserclient ports. at d
> >>> +\@failed to enumerate system datapaths: No such file or directory at d
> >>> +\@EAL:   Invalid NUMA socket, default to 0 at d
> >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> >>> unreliable clock cycles !@d
> >>> +\@EAL: No free hugepages reported in hugepages-1048576kB at d"])
> >>> +AT_CLEANUP
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>> +
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>> +dnl Ping vhost-user-client port
> >>> +AT_SETUP([OVS-DPDK datapath - ping vhost-user-client ports])
> >>> +AT_KEYWORDS([dpdk])
> >>> +OVS_DPDK_PRE_CHECK()
> >>> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> >>> +OVS_DPDK_START()
> >>> +
> >>> +dnl Add userspace bridge and attach it to OVS
> >>> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10
> >>> datapath_type=netdev])
> >>> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface
> >>> \
> >>> +          dpdkvhostuserclient0 \
> >>> +          type=dpdkvhostuserclient \
> >>> +          options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> >>> +         [stdout], [stderr])
> >>> +AT_CHECK([ovs-vsctl show], [], [stdout])
> >>> +
> >>> +dnl Parse log file
> >>> +AT_CHECK([grep "VHOST_CONFIG: vhost-user client: socket created"
> >>> ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "vHost User device 'dpdkvhostuserclient0' created in
> >>> 'client' mode, using client socket" ovs-vswitchd.log], [], [stdout])
> >>> +AT_CHECK([grep "VHOST_CONFIG: $OVS_RUNDIR/dpdkvhostclient0:
> >>> reconnecting..." ovs-vswitchd.log], [], [stdout])
> >>> +
> >>> +dnl Set up namespaces
> >>> +ADD_NAMESPACES(ns1, ns2)
> >>> +
> >>> +dnl Add veth device
> >>> +ADD_VETH(tap1, ns2, br10, "172.31.110.12/24")
> >>> +
> >>> +dnl Execute testpmd in background
> >>> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> >>> +tail -f /dev/null | testpmd --socket-mem=512 \
> >>> +
> >>> --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1"
> >>> \
> >>> +           --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> >>> +           --single-file-segments -- -a
> >>>> $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> >>> +
> >>> +dnl Give settling time to the testpmd processes - NOTE: this is bad
> >>> form.
> >>> +sleep 10
> >>> +
> >>> +dnl Move the tap devices to the namespaces
> >>> +AT_CHECK([ps aux | grep testpmd], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip link set tap0 netns ns1], [], [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link show | grep tap0], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip link set tap0 up], [], [stdout],
> >>> [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ip addr add 172.31.110.11/24 dev tap0], [],
> >>> +         [stdout], [stderr])
> >>> +
> >>> +AT_CHECK([ip netns exec ns1 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns2 ip link show], [], [stdout], [stderr])
> >>> +AT_CHECK([ip netns exec ns1 ping -c 4 -I tap0 172.31.110.12], [],
> >>> [stdout],
> >>> +         [stderr])
> >>> +
> >>> +dnl Clean up the testpmd now
> >>> +pkill -f -x -9 'tail -f /dev/null'
> >>> +
> >>> +dnl Clean up
> >>> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout],
> >>> [stderr])
> >>> +OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is
> >>> probably not loaded. at d
> >>> +\@Failed to enable flow control at d
> >>> +\@VHOST_CONFIG: recvmsg failed at d
> >>> +\@VHOST_CONFIG: failed to connect to $OVS_RUNDIR/dpdkvhostclient0: No
> >>> such
> >>> file or directory at d
> >>> +\@Global register is changed during at d
> >>> +\@dpdkvhostuser ports are considered deprecated;  please migrate to
> >>> dpdkvhostuserclient ports. at d
> >>> +\@failed to enumerate system datapaths: No such file or directory at d
> >>> +\@EAL:   Invalid NUMA socket, default to 0 at d
> >>> +\@EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using
> >>> unreliable clock cycles !@d
> >>> +\@EAL: No free hugepages reported in hugepages-1048576kB at d"])
> >>> +AT_CLEANUP
> >>> +dnl
> >>> --------------------------------------------------------------------------
> >>>
> >>
>

More information about the dev mailing list