[ovs-dev] 答复: 答复: [PATCH v4 0/3] Add support for TSO with DPDK

Yi Yang (杨燚)-云服务集团 yangyi01 at inspur.com
Sun Feb 23 08:45:16 UTC 2020


Hi, Flavio

Just let you know, your TSO support patch does need higher kernel version, it will be great if you can add document to tell users which kernel version is minimal requirement. I can confirm it can work after I used Ubuntu 18.04 and use kernel 5.3.0-40-generic.

vagrant at ubuntu1804:~$ uname -a
Linux ubuntu1804 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
vagrant at ubuntu1804:~$

By the way, TPACKET_V3 also can support TSO without any new change needed, so I think TPACKET_V3 can work normally with userspace-tso-enable=true as long as your TSO patch can work normally for veth-to-veth case.

I'll send out my tpacket patch v5 for review.

-----邮件原件-----
发件人: Yi Yang (杨燚)-云服务集团 
发送时间: 2020年2月23日 11:05
收件人: 'fbl at sysclose.org' <fbl at sysclose.org>
抄送: 'pkusunyifeng at gmail.com' <pkusunyifeng at gmail.com>; 'dev at openvswitch.org' <dev at openvswitch.org>; 'i.maximets at ovn.org' <i.maximets at ovn.org>; 'txfh2007 at aliyun.com' <txfh2007 at aliyun.com>
主题: 答复: 答复: [ovs-dev] [PATCH v4 0/3] Add support for TSO with DPDK
重要性: 高

Hi, Flavio

After I ran it repeatedly in different servers, I'm very sure it can't work on Ubuntu 16.04 kernel 4.15.0-55-generic and Upstream kernel 4.15.9, so can you tell me your kernel version when you ran run-iperf3.sh I provided ? I doubt this TSO patch for veth needs higher kernel version.

-----邮件原件-----
发件人: Flavio Leitner [mailto:fbl at sysclose.org] 
发送时间: 2020年2月20日 21:41
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com>
抄送: pkusunyifeng at gmail.com; dev at openvswitch.org; i.maximets at ovn.org; txfh2007 at aliyun.com
主题: Re: 答复: [ovs-dev] [PATCH v4 0/3] Add support for TSO with DPDK

On Thu, Feb 20, 2020 at 10:10:36AM +0000, Yi Yang (杨 D)-云服务集团 wrote:
> Hi, Flavio
> 
> I find this tso feature doesn't work normally on my Ubuntu 16.04, here 
> is my result. My kernel version is
> 
> $ uname -a
> Linux cmp008 4.15.0-55-generic #60~16.04.2-Ubuntu SMP Thu Jul 4 
> 09:03:09 UTC
> 2019 x86_64 x86_64 x86_64 GNU/Linux
> $

I tested with 4.15.0 upstream and it worked. Can you do the same?

> $ ./run-iperf3.sh
> Connecting to host 10.15.1.3, port 5201 [  4] local 10.15.1.2 port 
> 56466 connected to 10.15.1.3 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-10.00  sec  7.05 MBytes  5.91 Mbits/sec  2212   5.66 KBytes
> [  4]  10.00-20.00  sec  7.67 MBytes  6.44 Mbits/sec  2484   5.66 KBytes
> [  4]  20.00-30.00  sec  7.77 MBytes  6.52 Mbits/sec  2500   5.66 KBytes
> [  4]  30.00-40.00  sec  7.77 MBytes  6.52 Mbits/sec  2490   5.66 KBytes
> [  4]  40.00-50.00  sec  7.76 MBytes  6.51 Mbits/sec  2500   5.66 KBytes
> [  4]  50.00-60.00  sec  7.79 MBytes  6.54 Mbits/sec  2504   5.66 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-60.00  sec  45.8 MBytes  6.40 Mbits/sec  14690
> sender
> [  4]   0.00-60.00  sec  45.7 MBytes  6.40 Mbits/sec
> receiver

That looks like TSO packets are being dropped and the traffic is basically TCP retransmissions of MTU size.

fbl


> 
> Server output:
> Accepted connection from 10.15.1.2, port 56464 [  5] local 10.15.1.3 
> port 5201 connected to 10.15.1.2 port 56466
> [ ID] Interval           Transfer     Bandwidth
> [  5]   0.00-10.00  sec  6.90 MBytes  5.79 Mbits/sec
> [  5]  10.00-20.00  sec  7.71 MBytes  6.47 Mbits/sec [  5]  
> 20.00-30.00  sec  7.73 MBytes  6.48 Mbits/sec [  5]  30.00-40.00  sec  
> 7.79 MBytes  6.53 Mbits/sec [  5]  40.00-50.00  sec  7.79 MBytes  6.53 
> Mbits/sec [  5]  50.00-60.00  sec  7.79 MBytes  6.54 Mbits/sec
> 
> 
> iperf Done.
> $
> 
> But it does work for tap, I'm not sure if it is a kernel issue, which 
> kernel version are you using? I didn't use tpacket_v3 patch. Here is 
> my local ovs info.
> 
> $ git log
> commit 1223cf123ed141c0a0110ebed17572bdb2e3d0f4
> Author: Ilya Maximets <i.maximets at ovn.org>
> Date:   Thu Feb 6 14:24:23 2020 +0100
> 
>     netdev-dpdk: Don't enable offloading on HW device if not requested.
> 
>     DPDK drivers has different implementations of transmit functions.
>     Enabled offloading may cause driver to choose slower variant
>     significantly affecting performance if userspace TSO wasn't requested.
> 
>     Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload support")
>     Reported-by: David Marchand <david.marchand at redhat.com>
>     Acked-by: David Marchand <david.marchand at redhat.com>
>     Acked-by: Flavio Leitner <fbl at sysclose.org>
>     Acked-by: Kevin Traynor <ktraynor at redhat.com>
>     Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
> 
> commit 73858f9dbe83daf8cc8d4b604acc23eb62cc3f52
> Author: Flavio Leitner <fbl at sysclose.org>
> Date:   Mon Feb 3 18:45:50 2020 -0300
> 
>     netdev-linux: Prepend the std packet in the TSO packet
> 
>     Usually TSO packets are close to 50k, 60k bytes long, so to
>     to copy less bytes when receiving a packet from the kernel
>     change the approach. Instead of extending the MTU sized
>     packet received and append with remaining TSO data from
>     the TSO buffer, allocate a TSO packet with enough headroom
>     to prepend the std packet data.
> 
>     Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload support")
>     Suggested-by: Ben Pfaff <blp at ovn.org>
>     Signed-off-by: Flavio Leitner <fbl at sysclose.org>
>     Signed-off-by: Ben Pfaff <blp at ovn.org>
> 
> commit 2297cbe6cc25b6b1862c499ce8f16f52f75d9e5f
> Author: Flavio Leitner <fbl at sysclose.org>
> Date:   Mon Feb 3 11:22:22 2020 -0300
> 
>     netdev-linux-private: fix max length to be 16 bits
> 
>     The dp_packet length is limited to 16 bits, so document that
>     and fix the length value accordingly.
> 
>     Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload support")
>     Signed-off-by: Flavio Leitner <fbl at sysclose.org>
>     Signed-off-by: Ben Pfaff <blp at ovn.org>
> 
> commit 3d6a6f450af5b7eaf4b532983cb14458ae792b72
> Author: David Marchand <david.marchand at redhat.com>
> Date:   Tue Feb 4 22:28:26 2020 +0100
> 
>     netdev-dpdk: Fix port init when lacking Tx offloads for TSO.
> 
>     The check on TSO capability did not ensure ip checksum, tcp checksum and
>     TSO tx offloads were available which resulted in a port init failure
>     (example below with a ena device):
> 
>     *2020-02-04T17:42:52.976Z|00084|dpdk|ERR|Ethdev port_id=0 requested Tx
>     offloads 0x2a doesn't match Tx offloads capabilities 0xe in
>     rte_eth_dev_configure()*
> 
>     Fixes: 29cf9c1b3b9c ("userspace: Add TCP Segmentation Offload 
> support")
> 
>     Reported-by: Ravi Kerur <rkerur at gmail.com>
>     Signed-off-by: David Marchand <david.marchand at redhat.com>
>     Acked-by: Kevin Traynor <ktraynor at redhat.com>
>     Acked-by: Flavio Leitner <fbl at sysclose.org>
>     Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
> 
> commit 8e371aa497aa95e3562d53f566c2d634b4b0f589
> Author: Kirill A. Kornilov <kirill at tp>
> Date:   Mon Jan 13 12:29:10 2020 +0300
> 
>     vswitchd: Add serial number configuration.
> 
>     Signed-off-by: Kirill A. Kornilov <kornilov at zelax.ru>
>     Signed-off-by: Ben Pfaff <blp at ovn.org>
> 
> I applied your tap patch.
> 
> $ git diff
> diff --git a/lib/netdev-linux.c b/lib/netdev-linux.c index 
> c6f3d27..74a5728 100644
> --- a/lib/netdev-linux.c
> +++ b/lib/netdev-linux.c
> @@ -1010,6 +1010,23 @@ netdev_linux_construct_tap(struct netdev *netdev_)
>          goto error_close;
>      }
> 
> +    if (userspace_tso_enabled()) {
> +        /* Old kernels don't support TUNSETOFFLOAD. If TUNSETOFFLOAD is
> +         * available, it will return EINVAL when a flag is unknown.
> +         * Therefore, try enabling offload with no flags to check
> +         * if TUNSETOFFLOAD support is available or not. */
> +        if (ioctl(netdev->tap_fd, TUNSETOFFLOAD, 0) == 0 || errno !=
> EINVAL) {
> +            unsigned long oflags = TUN_F_CSUM | TUN_F_TSO4 | 
> + TUN_F_TSO6;
> +
> +            if (ioctl(netdev->tap_fd, TUNSETOFFLOAD, oflags) == -1) {
> +                VLOG_WARN("%s: enabling tap offloading failed: %s", name,
> +                          ovs_strerror(errno));
> +                error = errno;
> +                goto error_close;
> +            }
> +        }
> +    }
> +
>      netdev->present = true;
>      return 0;
> 
> $
> 
> Here is performance result for tap.
> 
> $ ./run-iperf3.sh
> Connecting to host 10.15.1.3, port 5201 [  4] local 10.15.1.2 port 
> 56480 connected to 10.15.1.3 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-10.00  sec  19.4 GBytes  16.7 Gbits/sec    0   3.05 MBytes
> [  4]  10.00-20.00  sec  18.3 GBytes  15.7 Gbits/sec    0   3.05 MBytes
> [  4]  20.00-30.00  sec  17.6 GBytes  15.1 Gbits/sec    0   3.05 MBytes
> [  4]  30.00-40.00  sec  19.3 GBytes  16.6 Gbits/sec    0   3.05 MBytes
> [  4]  40.00-50.00  sec  18.8 GBytes  16.1 Gbits/sec    0   3.05 MBytes
> [  4]  50.00-60.00  sec  17.9 GBytes  15.4 Gbits/sec    0   3.05 MBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-60.00  sec   111 GBytes  15.9 Gbits/sec    0             sender
> [  4]   0.00-60.00  sec   111 GBytes  15.9 Gbits/sec
> receiver
> 
> Server output:
> Accepted connection from 10.15.1.2, port 56478 [  5] local 10.15.1.3 
> port 5201 connected to 10.15.1.2 port 56480
> [ ID] Interval           Transfer     Bandwidth
> [  5]   0.00-10.00  sec  19.3 GBytes  16.6 Gbits/sec
> [  5]  10.00-20.00  sec  18.3 GBytes  15.7 Gbits/sec [  5]  
> 20.00-30.00  sec  17.6 GBytes  15.1 Gbits/sec [  5]  30.00-40.00  sec  
> 19.3 GBytes  16.6 Gbits/sec [  5]  40.00-50.00  sec  18.8 GBytes  16.1 
> Gbits/sec [  5]  50.00-60.00  sec  17.9 GBytes  15.4 Gbits/sec
> 
> 
> iperf Done.
> $



--
fbl


More information about the dev mailing list