[ovs-dev] [PATCH] system-dpdk: add negotiation check for userspace-tso
Flavio Leitner
fbl at sysclose.org
Fri Jul 3 00:42:48 UTC 2020
Hi Gowrishankar,
Thanks for working on this patch, it's a good idea to test the
TSO negotiation.
Which dpdk version are you testing? Because I am getting the same
results on 19.11.3 or master. See the details below.
Sometimes it fails with this in testpmd-dpdkvhostuser0.log:
EAL: Detected 4 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Cannot create lock on '/var/run/dpdk/page0/config'. Is another
primary process running?
EAL: FATAL: Cannot init config
EAL: Cannot init config
EAL: Error - exiting with code: 1
Cause: Cannot init EAL: Success
Because the previous testpmd hasn't terminated fully yet.
I added this hack and it helped:
@@ -302,6 +302,7 @@ AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
dnl Clean up the testpmd now
pkill -f -x -9 'tail -f /dev/null'
+sleep 1
dnl Add vhostuser port (server mode)
AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
It also fails regularly checking for TCP_TSO flag in testpmd output.
I did a copy of the log right before checking and compared with the
log at the end of the test. Turns out that the log is not updated yet.
So, I improved the hack above to terminate testpmd earlier and give
extra time in both places:
@@ -287,6 +287,9 @@ tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
dnl Give settling time to the testpmd processes - NOTE: this is bad form.
sleep 10
+dnl Clean up the testpmd now
+pkill -f -x -9 'tail -f /dev/null'
+sleep 1
dnl Check whether TSO is turned on (host side)
AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
@@ -300,9 +303,6 @@ AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END
dnl Clean up
AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
-dnl Clean up the testpmd now
-pkill -f -x -9 'tail -f /dev/null'
-
dnl Add vhostuser port (server mode)
AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
type=dpdkvhostuser], [],
Same change for the next chunk below that. Perhaps it can wait until
the /var/run/dpdk/page0 is gone.
Then it fails when testing vhost in server mode because OVS doesn't
pass the EXTBUF flag, so TSO gets disabled:
@@ -1349,6 +1349,12 @@ netdev_dpdk_vhost_construct(struct netdev *netdev)
/* There is no support for multi-segments buffers. */
dev->vhost_driver_flags |= RTE_VHOST_USER_LINEARBUF_SUPPORT;
+
+ /* Enable External Buffers if TCP Segmentation Offload is enabled. */
+ if (userspace_tso_enabled()) {
+ dev->vhost_driver_flags |= RTE_VHOST_USER_EXTBUF_SUPPORT;
+ }
+
err = rte_vhost_driver_register(dev->vhost_id, dev->vhost_driver_flags);
if (err) {
VLOG_ERR("vhost-user socket device setup failure for socket %s\n",
I am going to post that as a patch tomorrow.
Now it always runs until vswitchd is stopped, but then it fails because
vswitchd reports a NUMA socket error:
2020-07-03T00:11:35.575Z|00019|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
+2020-07-03T00:11:35.575Z|00021|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
+2020-07-03T00:11:35.652Z|00054|dpdk|WARN|VHOST_CONFIG: failed to connect to /home/fleitner/repo/ovs/tests/system-dpdk-testsuite.dir/6/dpdkvhostclient0: No such file or directory
+2020-07-03T00:11:45.764Z|00044|dpdk|WARN|VHOST_CONFIG: failed to connect to /home/fleitner/repo/ovs/tests/system-dpdk-testsuite.dir/6/dpdkvhostclient0: No such file or directory
+2020-07-03T00:11:46.801Z|00065|netdev_dpdk|WARN|dpdkvhostuser ports are considered deprecated; please migrate to dpdkvhostuserclient ports.
+2020-07-03T00:11:35Z|00019|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
+2020-07-03T00:11:35Z|00021|dpdk|WARN|EAL: Invalid NUMA socket, default to 0
My system is a virtual machine running fedora 32 with a virtio NIC
for testing. It has selinux disabled, upstream kernel cd77006e01b
recompiled with fedora's kernel config + CONFIG_VFIO_NOIOMMU=y to
load vfio kernel module passing enable_unsafe_noiommu_mode=1.
fbl
On Wed, Jun 24, 2020 at 08:10:58PM +0530, Gowrishankar Muthukrishnan wrote:
> This patch adds minimal check for userspace-tso in system-dpdk
> tests, starting with verification on virtio negotiation.
>
> Signed-off-by: Gowrishankar Muthukrishnan <gmuthukr at redhat.com>
> ---
> v4:
> - configure offload settings in testpmd
> ---
> tests/system-dpdk-macros.at | 17 +++-
> tests/system-dpdk.at | 228 +++++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 236 insertions(+), 9 deletions(-)
>
> diff --git a/tests/system-dpdk-macros.at b/tests/system-dpdk-macros.at
> index c6708ca..9e55f10 100644
> --- a/tests/system-dpdk-macros.at
> +++ b/tests/system-dpdk-macros.at
> @@ -33,13 +33,11 @@ m4_define([OVS_DPDK_PRE_PHY_SKIP],
> ])
>
>
> -# OVS_DPDK_START()
> +# OVS_DB_START()
> #
> -# Create an empty database and start ovsdb-server. Add special configuration
> -# dpdk-init to enable DPDK functionality. Start ovs-vswitchd connected to that
> -# database using system devices (no dummies).
> +# Create an empty database and start ovsdb-server.
> #
> -m4_define([OVS_DPDK_START],
> +m4_define([OVS_DB_START],
> [dnl Create database.
> AT_CHECK([touch .conf.db.~lock~])
> AT_CHECK([ovsdb-tool create conf.db $abs_top_srcdir/vswitchd/vswitch.ovsschema])
> @@ -54,7 +52,16 @@ m4_define([OVS_DPDK_START],
>
> dnl Initialize database.
> AT_CHECK([ovs-vsctl --no-wait init])
> +])
> +
>
> +# OVS_DPDK_START()
> +#
> +# Add special configuration dpdk-init to enable DPDK functionality.
> +# Start ovs-vswitchd connected to that database using system devices (no dummies).
> +#
> +m4_define([OVS_DPDK_START],
> + [
> dnl Enable DPDK functionality
> AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true])
>
> diff --git a/tests/system-dpdk.at b/tests/system-dpdk.at
> index a015d52..ba9a1e7 100644
> --- a/tests/system-dpdk.at
> +++ b/tests/system-dpdk.at
> @@ -1,6 +1,11 @@
> m4_define([CONFIGURE_VETH_OFFLOADS],
> [AT_CHECK([ethtool -K $1 tx off], [0], [ignore], [ignore])])
>
> +m4_define([SET_NUMA_NODE],
> + [
> + AT_CHECK([lscpu | awk '/NUMA node\(s\)/ {c=1; while (c++<$(3)) {printf "$1,"}; print "$1"}' > NUMA_NODE])
> +])
> +
> AT_BANNER([OVS-DPDK unit tests])
>
> dnl --------------------------------------------------------------------------
> @@ -8,6 +13,7 @@ dnl Check if EAL init is successful
> AT_SETUP([OVS-DPDK - EAL init])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> +OVS_DB_START()
> OVS_DPDK_START()
> AT_CHECK([grep "DPDK Enabled - initializing..." ovs-vswitchd.log], [], [stdout])
> AT_CHECK([grep "EAL" ovs-vswitchd.log], [], [stdout])
> @@ -27,6 +33,7 @@ AT_SETUP([OVS-DPDK - add standard DPDK port])
> AT_KEYWORDS([dpdk])
>
> OVS_DPDK_PRE_PHY_SKIP()
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Add userspace bridge and attach it to OVS
> @@ -53,6 +60,7 @@ dnl Add vhost-user-client port
> AT_SETUP([OVS-DPDK - add vhost-user-client port])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Add userspace bridge and attach it to OVS
> @@ -86,11 +94,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user ports])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Find number of sockets
> -AT_CHECK([lscpu], [], [stdout])
> -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE])
> +SET_NUMA_NODE([512])
>
> dnl Add userspace bridge and attach it to OVS
> AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> @@ -163,11 +171,11 @@ AT_SETUP([OVS-DPDK - ping vhost-user-client ports])
> AT_KEYWORDS([dpdk])
> OVS_DPDK_PRE_CHECK()
> AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> OVS_DPDK_START()
>
> dnl Find number of sockets
> -AT_CHECK([lscpu], [], [stdout])
> -AT_CHECK([cat stdout | grep "NUMA node(s)" | awk '{c=1; while (c++<$(3)) {printf "512,"}; print "512"}' > NUMA_NODE])
> +SET_NUMA_NODE([512])
>
> dnl Add userspace bridge and attach it to OVS
> AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> @@ -232,3 +240,215 @@ OVS_VSWITCHD_STOP(["\@does not exist. The Open vSwitch kernel module is probably
> \@EAL: No free hugepages reported in hugepages-1048576kB at d"])
> AT_CLEANUP
> dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl validate tso negotiation (with userspace-tso)
> +AT_SETUP([OVS-DPDK - validate tso negotiation (with userspace-tso)])
> +AT_KEYWORDS([dpdk])
> +OVS_DPDK_PRE_CHECK()
> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:userspace-tso-enable=true])
> +OVS_DPDK_START()
> +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support enabled' \
> + ovs-vswitchd.log],[ignore],[dnl
> +1
> +])
> +dnl Find number of sockets
> +SET_NUMA_NODE([512])
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +
> +dnl Add vhostuser port (client mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \
> + dpdkvhostuserclient0 \
> + type=dpdkvhostuserclient \
> + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned on (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[1801])
> +
> +dnl Check whether TSO is turned on (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[1])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Add vhostuser port (server mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
> + type=dpdkvhostuser], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned on (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[1801])
> +
> +dnl Check whether TSO is turned on (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[1])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP()
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> +
> +dnl --------------------------------------------------------------------------
> +dnl validate tso negotiation (without userspace-tso)
> +AT_SETUP([OVS-DPDK - validate tso negotiation (without userspace-tso)])
> +AT_KEYWORDS([dpdk])
> +OVS_DPDK_PRE_CHECK()
> +AT_SKIP_IF([! which testpmd >/dev/null 2>/dev/null])
> +OVS_DB_START()
> +AT_CHECK([ovs-vsctl --no-wait set Open_vSwitch . other_config:userspace-tso-enable=false])
> +OVS_DPDK_START()
> +AT_CHECK([grep -c 'Userspace TCP Segmentation Offloading support enabled' \
> + ovs-vswitchd.log],[ignore],[dnl
> +0
> +])
> +dnl Find number of sockets
> +SET_NUMA_NODE([512])
> +
> +dnl Add userspace bridge and attach it to OVS
> +AT_CHECK([ovs-vsctl add-br br10 -- set bridge br10 datapath_type=netdev])
> +
> +dnl Add vhostuser port (client mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuserclient0 -- set Interface \
> + dpdkvhostuserclient0 \
> + type=dpdkvhostuserclient \
> + options:vhost-server-path=$OVS_RUNDIR/dpdkvhostclient0], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostclient0,server=1" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned off (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[0])
> +
> +dnl Check whether TSO is turned off (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuserclient0.log],[0],[0])
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuserclient0], [], [stdout], [stderr])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Add vhostuser port (server mode)
> +AT_CHECK([ovs-vsctl add-port br10 dpdkvhostuser0 -- set Interface dpdkvhostuser0 \
> + type=dpdkvhostuser], [],
> + [stdout], [stderr])
> +AT_CHECK([ovs-vsctl show], [], [stdout])
> +
> +dnl Execute testpmd in background
> +on_exit "pkill -f -x -9 'tail -f /dev/null'"
> +AT_CHECK([echo "show device info all" > CMDFILE])
> +AT_CHECK([echo "stop" >> CMDFILE])
> +AT_CHECK([echo "port stop 0" >> CMDFILE])
> +AT_CHECK([echo "tso set 1500 0" >> CMDFILE])
> +AT_CHECK([echo "csum set tcp hw 0" >> CMDFILE])
> +AT_CHECK([echo "port start 0" >> CMDFILE])
> +AT_CHECK([echo "start" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload capabilities" >> CMDFILE])
> +AT_CHECK([echo "show port 0 tx_offload configuration" >> CMDFILE])
> +tail -f /dev/null | testpmd --socket-mem="$(cat NUMA_NODE)" --no-pci\
> + --vdev="net_virtio_user,path=$OVS_RUNDIR/dpdkvhostuser0" \
> + --vdev="net_tap0,iface=tap0" --file-prefix page0 \
> + --single-file-segments -- --cmdline-file=CMDFILE \
> + -a >$OVS_RUNDIR/testpmd-dpdkvhostuser0.log 2>&1 &
> +
> +dnl Give settling time to the testpmd processes - NOTE: this is bad form.
> +sleep 10
> +
> +dnl Check whether TSO is turned off (host side)
> +AT_CHECK([awk '/negotiated Virtio features/ {a=$NF} END{print a}' \
> + $OVS_RUNDIR/ovs-vswitchd.log],[],[stdout])
> +AT_CHECK([printf "%X" $(( $(cat stdout) & ((1<<0)|(1<<11)|(1<<12)) ))],[],[0])
> +
> +dnl Check whether TSO is turned off (guest side)
> +AT_CHECK([awk 'BEGIN{n=0} /Per Port/ && /(TCP|UDP)_CKSUM/ && /TCP_TSO/ {n++} END{printf n}' \
> + $OVS_RUNDIR/testpmd-dpdkvhostuser0.log],[0],[0])
> +
> +dnl Clean up the testpmd now
> +pkill -f -x -9 'tail -f /dev/null'
> +
> +dnl Clean up
> +AT_CHECK([ovs-vsctl del-port br10 dpdkvhostuser0], [], [stdout], [stderr])
> +OVS_VSWITCHD_STOP()
> +AT_CLEANUP
> +dnl --------------------------------------------------------------------------
> --
> 1.8.3.1
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
--
fbl
More information about the dev
mailing list