[ovs-dev] 答复: 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath

Yi Yang (杨燚)-云服务集团 yangyi01 at inspur.com
Mon Apr 13 09:02:24 UTC 2020


Hi, William

As you have known, I sent out a tpacket_v3 kernel side patch to net-next, tpacket maintainer worries it can impact on other use cases, so he hopes we can use TPACKET_V2 for TSO, I tried TPACKET_V2 for TSO, it is indeed a good choice in case that tpacket_v3 in kernel side isn't ready for this.

I have sent out v8 http://patchwork.ozlabs.org/project/openvswitch/patch/20200413083905.11128-1-yang_y_yi@163.com/ for this.

Per my test, its performance is much better than recvmmsg/sendmmsg in case of TSO, here is my test results for your reference.

[yangyi at localhost ovs-master]$ uname -a
Linux localhost.localdomain 5.5.9+ #40 SMP Mon Mar 30 05:54:05 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
recvmmsg & sendmmsg TSO
=======================
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 43354 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  5.64 GBytes  4.84 Gbits/sec  48224    198 KBytes
[  4]  10.00-20.00  sec  5.59 GBytes  4.80 Gbits/sec  46100    182 KBytes
[  4]  20.00-30.00  sec  5.68 GBytes  4.88 Gbits/sec  48959    226 KBytes
[  4]  30.00-40.00  sec  5.58 GBytes  4.80 Gbits/sec  49035    161 KBytes
[  4]  40.00-50.00  sec  5.54 GBytes  4.76 Gbits/sec  49306    256 KBytes
[  4]  50.00-60.00  sec  5.58 GBytes  4.80 Gbits/sec  48558    197 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  33.6 GBytes  4.81 Gbits/sec  290182             sender
[  4]   0.00-60.00  sec  33.6 GBytes  4.81 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 43352
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43354
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  5.64 GBytes  4.84 Gbits/sec
[  5]  10.00-20.00  sec  5.59 GBytes  4.80 Gbits/sec
[  5]  20.00-30.00  sec  5.68 GBytes  4.88 Gbits/sec
[  5]  30.00-40.00  sec  5.58 GBytes  4.80 Gbits/sec
[  5]  40.00-50.00  sec  5.54 GBytes  4.76 Gbits/sec
[  5]  50.00-60.00  sec  5.58 GBytes  4.80 Gbits/sec
[  5]  60.00-60.00  sec   255 KBytes  4.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-60.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-60.00  sec  33.6 GBytes  4.81 Gbits/sec                  receiver


iperf Done.
[yangyi at localhost ovs-master]$

TPACKET_V2 TSO
==============
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh                                Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 41600 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  10.6 GBytes  9.11 Gbits/sec   23   3.00 MBytes
[  4]  10.00-20.00  sec  10.6 GBytes  9.13 Gbits/sec    0   3.00 MBytes
[  4]  20.00-30.00  sec  10.7 GBytes  9.15 Gbits/sec    0   3.00 MBytes
[  4]  30.00-40.00  sec  10.6 GBytes  9.13 Gbits/sec   32   3.00 MBytes
[  4]  40.00-50.00  sec  10.7 GBytes  9.17 Gbits/sec    0   3.00 MBytes
[  4]  50.00-60.00  sec  10.5 GBytes  9.06 Gbits/sec    0   3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  63.7 GBytes  9.12 Gbits/sec   55             sender
[  4]   0.00-60.00  sec  63.7 GBytes  9.12 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 41598
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 41600
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  10.6 GBytes  9.11 Gbits/sec
[  5]  10.00-20.00  sec  10.6 GBytes  9.13 Gbits/sec
[  5]  20.00-30.00  sec  10.7 GBytes  9.15 Gbits/sec
[  5]  30.00-40.00  sec  10.6 GBytes  9.13 Gbits/sec
[  5]  40.00-50.00  sec  10.7 GBytes  9.17 Gbits/sec
[  5]  50.00-60.00  sec  10.5 GBytes  9.06 Gbits/sec
[  5]  60.00-60.00  sec  2.72 MBytes  15.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-60.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-60.00  sec  63.7 GBytes  9.12 Gbits/sec                  receiver


iperf Done.
[yangyi at localhost ovs-master]$

I also tested it by using Ubuntu 4.15 kernel, it have better performance.

[yangyi at localhost ovs-master]$ uname -a
Linux localhost.localdomain 4.15.18 #10 SMP Wed Mar 25 06:02:27 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[yangyi at localhost ovs-master]$

recvmmsg & sendmmsg TSO
=======================
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 59550 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  6.28 GBytes  5.39 Gbits/sec  56673    171 KBytes
[  4]  10.00-20.00  sec  6.41 GBytes  5.50 Gbits/sec  56704    184 KBytes
[  4]  20.00-30.00  sec  6.64 GBytes  5.71 Gbits/sec  55720    189 KBytes
[  4]  30.00-40.00  sec  6.52 GBytes  5.60 Gbits/sec  53433    178 KBytes
[  4]  40.00-50.00  sec  6.41 GBytes  5.51 Gbits/sec  52541    185 KBytes
[  4]  50.00-60.00  sec  6.52 GBytes  5.60 Gbits/sec  56081    141 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  38.8 GBytes  5.55 Gbits/sec  331152             sender
[  4]   0.00-60.00  sec  38.8 GBytes  5.55 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 59548
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 59550
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  6.25 GBytes  5.37 Gbits/sec
[  5]  10.00-20.00  sec  6.41 GBytes  5.51 Gbits/sec
[  5]  20.00-30.00  sec  6.64 GBytes  5.71 Gbits/sec
[  5]  30.00-40.00  sec  6.52 GBytes  5.60 Gbits/sec
[  5]  40.00-50.00  sec  6.41 GBytes  5.51 Gbits/sec
[  5]  50.00-60.00  sec  6.51 GBytes  5.60 Gbits/sec
[  5]  60.00-60.04  sec  22.5 MBytes  4.71 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-60.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-60.04  sec  38.8 GBytes  5.55 Gbits/sec                  receiver


iperf Done.
[yangyi at localhost ovs-master]$

TPACKET_V2 TSO
===================
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 32884 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  10.7 GBytes  9.21 Gbits/sec    7   3.13 MBytes
[  4]  10.00-20.00  sec  10.8 GBytes  9.25 Gbits/sec    0   3.13 MBytes
[  4]  20.00-30.00  sec  10.8 GBytes  9.25 Gbits/sec    0   3.13 MBytes
[  4]  30.00-40.00  sec  10.8 GBytes  9.29 Gbits/sec    0   3.13 MBytes
[  4]  40.00-50.00  sec  10.8 GBytes  9.30 Gbits/sec    0   3.13 MBytes
[  4]  50.00-60.00  sec  10.7 GBytes  9.20 Gbits/sec    0   3.13 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  64.6 GBytes  9.25 Gbits/sec    7             sender
[  4]   0.00-60.00  sec  64.6 GBytes  9.25 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 32882
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 32884
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  10.7 GBytes  9.17 Gbits/sec
[  5]  10.00-20.00  sec  10.8 GBytes  9.25 Gbits/sec
[  5]  20.00-30.00  sec  10.8 GBytes  9.25 Gbits/sec
[  5]  30.00-40.00  sec  10.8 GBytes  9.29 Gbits/sec
[  5]  40.00-50.00  sec  10.8 GBytes  9.30 Gbits/sec
[  5]  50.00-60.00  sec  10.7 GBytes  9.20 Gbits/sec
[  5]  60.00-60.04  sec  46.0 MBytes  9.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-60.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-60.04  sec  64.6 GBytes  9.24 Gbits/sec                  receiver


iperf Done.
[yangyi at localhost ovs-master]$

-----邮件原件-----
发件人: William Tu [mailto:u9012063 at gmail.com] 
发送时间: 2020年3月19日 22:53
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com>
抄送: i.maximets at ovn.org; yang_y_yi at 163.com; ovs-dev at openvswitch.org
主题: Re: [ovs-dev] 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath

On Wed, Mar 18, 2020 at 8:12 PM Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com> wrote:
>
> Hi, folks
>
> As I said, TPACKET_V3 does have kernel implementation issue, I tried to fix it in Linux kernel 5.5.9, here is my test data with tpacket_v3 and tso enabled. On my low end server, my goal is to reach 16Gbps at least, I still have another idea to improve it.
>
Can you share your kernel fix?
Or have you sent patch somewhere?
William


More information about the dev mailing list