[ovs-dev] 答复: 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath
Yi Yang (杨燚)-云服务集团
yangyi01 at inspur.com
Mon Apr 13 09:02:24 UTC 2020
Hi, William
As you have known, I sent out a tpacket_v3 kernel side patch to net-next, tpacket maintainer worries it can impact on other use cases, so he hopes we can use TPACKET_V2 for TSO, I tried TPACKET_V2 for TSO, it is indeed a good choice in case that tpacket_v3 in kernel side isn't ready for this.
I have sent out v8 http://patchwork.ozlabs.org/project/openvswitch/patch/20200413083905.11128-1-yang_y_yi@163.com/ for this.
Per my test, its performance is much better than recvmmsg/sendmmsg in case of TSO, here is my test results for your reference.
[yangyi at localhost ovs-master]$ uname -a
Linux localhost.localdomain 5.5.9+ #40 SMP Mon Mar 30 05:54:05 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
recvmmsg & sendmmsg TSO
=======================
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[ 4] local 10.15.1.2 port 43354 connected to 10.15.1.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-10.00 sec 5.64 GBytes 4.84 Gbits/sec 48224 198 KBytes
[ 4] 10.00-20.00 sec 5.59 GBytes 4.80 Gbits/sec 46100 182 KBytes
[ 4] 20.00-30.00 sec 5.68 GBytes 4.88 Gbits/sec 48959 226 KBytes
[ 4] 30.00-40.00 sec 5.58 GBytes 4.80 Gbits/sec 49035 161 KBytes
[ 4] 40.00-50.00 sec 5.54 GBytes 4.76 Gbits/sec 49306 256 KBytes
[ 4] 50.00-60.00 sec 5.58 GBytes 4.80 Gbits/sec 48558 197 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 33.6 GBytes 4.81 Gbits/sec 290182 sender
[ 4] 0.00-60.00 sec 33.6 GBytes 4.81 Gbits/sec receiver
Server output:
Accepted connection from 10.15.1.2, port 43352
[ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43354
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 5.64 GBytes 4.84 Gbits/sec
[ 5] 10.00-20.00 sec 5.59 GBytes 4.80 Gbits/sec
[ 5] 20.00-30.00 sec 5.68 GBytes 4.88 Gbits/sec
[ 5] 30.00-40.00 sec 5.58 GBytes 4.80 Gbits/sec
[ 5] 40.00-50.00 sec 5.54 GBytes 4.76 Gbits/sec
[ 5] 50.00-60.00 sec 5.58 GBytes 4.80 Gbits/sec
[ 5] 60.00-60.00 sec 255 KBytes 4.15 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.00 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.00 sec 33.6 GBytes 4.81 Gbits/sec receiver
iperf Done.
[yangyi at localhost ovs-master]$
TPACKET_V2 TSO
==============
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh Connecting to host 10.15.1.3, port 5201
[ 4] local 10.15.1.2 port 41600 connected to 10.15.1.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-10.00 sec 10.6 GBytes 9.11 Gbits/sec 23 3.00 MBytes
[ 4] 10.00-20.00 sec 10.6 GBytes 9.13 Gbits/sec 0 3.00 MBytes
[ 4] 20.00-30.00 sec 10.7 GBytes 9.15 Gbits/sec 0 3.00 MBytes
[ 4] 30.00-40.00 sec 10.6 GBytes 9.13 Gbits/sec 32 3.00 MBytes
[ 4] 40.00-50.00 sec 10.7 GBytes 9.17 Gbits/sec 0 3.00 MBytes
[ 4] 50.00-60.00 sec 10.5 GBytes 9.06 Gbits/sec 0 3.00 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 63.7 GBytes 9.12 Gbits/sec 55 sender
[ 4] 0.00-60.00 sec 63.7 GBytes 9.12 Gbits/sec receiver
Server output:
Accepted connection from 10.15.1.2, port 41598
[ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 41600
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 10.6 GBytes 9.11 Gbits/sec
[ 5] 10.00-20.00 sec 10.6 GBytes 9.13 Gbits/sec
[ 5] 20.00-30.00 sec 10.7 GBytes 9.15 Gbits/sec
[ 5] 30.00-40.00 sec 10.6 GBytes 9.13 Gbits/sec
[ 5] 40.00-50.00 sec 10.7 GBytes 9.17 Gbits/sec
[ 5] 50.00-60.00 sec 10.5 GBytes 9.06 Gbits/sec
[ 5] 60.00-60.00 sec 2.72 MBytes 15.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.00 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.00 sec 63.7 GBytes 9.12 Gbits/sec receiver
iperf Done.
[yangyi at localhost ovs-master]$
I also tested it by using Ubuntu 4.15 kernel, it have better performance.
[yangyi at localhost ovs-master]$ uname -a
Linux localhost.localdomain 4.15.18 #10 SMP Wed Mar 25 06:02:27 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
[yangyi at localhost ovs-master]$
recvmmsg & sendmmsg TSO
=======================
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[ 4] local 10.15.1.2 port 59550 connected to 10.15.1.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-10.00 sec 6.28 GBytes 5.39 Gbits/sec 56673 171 KBytes
[ 4] 10.00-20.00 sec 6.41 GBytes 5.50 Gbits/sec 56704 184 KBytes
[ 4] 20.00-30.00 sec 6.64 GBytes 5.71 Gbits/sec 55720 189 KBytes
[ 4] 30.00-40.00 sec 6.52 GBytes 5.60 Gbits/sec 53433 178 KBytes
[ 4] 40.00-50.00 sec 6.41 GBytes 5.51 Gbits/sec 52541 185 KBytes
[ 4] 50.00-60.00 sec 6.52 GBytes 5.60 Gbits/sec 56081 141 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 38.8 GBytes 5.55 Gbits/sec 331152 sender
[ 4] 0.00-60.00 sec 38.8 GBytes 5.55 Gbits/sec receiver
Server output:
Accepted connection from 10.15.1.2, port 59548
[ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 59550
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 6.25 GBytes 5.37 Gbits/sec
[ 5] 10.00-20.00 sec 6.41 GBytes 5.51 Gbits/sec
[ 5] 20.00-30.00 sec 6.64 GBytes 5.71 Gbits/sec
[ 5] 30.00-40.00 sec 6.52 GBytes 5.60 Gbits/sec
[ 5] 40.00-50.00 sec 6.41 GBytes 5.51 Gbits/sec
[ 5] 50.00-60.00 sec 6.51 GBytes 5.60 Gbits/sec
[ 5] 60.00-60.04 sec 22.5 MBytes 4.71 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.04 sec 38.8 GBytes 5.55 Gbits/sec receiver
iperf Done.
[yangyi at localhost ovs-master]$
TPACKET_V2 TSO
===================
[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[ 4] local 10.15.1.2 port 32884 connected to 10.15.1.3 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-10.00 sec 10.7 GBytes 9.21 Gbits/sec 7 3.13 MBytes
[ 4] 10.00-20.00 sec 10.8 GBytes 9.25 Gbits/sec 0 3.13 MBytes
[ 4] 20.00-30.00 sec 10.8 GBytes 9.25 Gbits/sec 0 3.13 MBytes
[ 4] 30.00-40.00 sec 10.8 GBytes 9.29 Gbits/sec 0 3.13 MBytes
[ 4] 40.00-50.00 sec 10.8 GBytes 9.30 Gbits/sec 0 3.13 MBytes
[ 4] 50.00-60.00 sec 10.7 GBytes 9.20 Gbits/sec 0 3.13 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-60.00 sec 64.6 GBytes 9.25 Gbits/sec 7 sender
[ 4] 0.00-60.00 sec 64.6 GBytes 9.25 Gbits/sec receiver
Server output:
Accepted connection from 10.15.1.2, port 32882
[ 5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 32884
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-10.00 sec 10.7 GBytes 9.17 Gbits/sec
[ 5] 10.00-20.00 sec 10.8 GBytes 9.25 Gbits/sec
[ 5] 20.00-30.00 sec 10.8 GBytes 9.25 Gbits/sec
[ 5] 30.00-40.00 sec 10.8 GBytes 9.29 Gbits/sec
[ 5] 40.00-50.00 sec 10.8 GBytes 9.30 Gbits/sec
[ 5] 50.00-60.00 sec 10.7 GBytes 9.20 Gbits/sec
[ 5] 60.00-60.04 sec 46.0 MBytes 9.29 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-60.04 sec 0.00 Bytes 0.00 bits/sec sender
[ 5] 0.00-60.04 sec 64.6 GBytes 9.24 Gbits/sec receiver
iperf Done.
[yangyi at localhost ovs-master]$
-----邮件原件-----
发件人: William Tu [mailto:u9012063 at gmail.com]
发送时间: 2020年3月19日 22:53
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com>
抄送: i.maximets at ovn.org; yang_y_yi at 163.com; ovs-dev at openvswitch.org
主题: Re: [ovs-dev] 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath
On Wed, Mar 18, 2020 at 8:12 PM Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com> wrote:
>
> Hi, folks
>
> As I said, TPACKET_V3 does have kernel implementation issue, I tried to fix it in Linux kernel 5.5.9, here is my test data with tpacket_v3 and tso enabled. On my low end server, my goal is to reach 16Gbps at least, I still have another idea to improve it.
>
Can you share your kernel fix?
Or have you sent patch somewhere?
William
More information about the dev
mailing list