[ovs-dev] 答复: 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath

Yi Yang (杨燚)-云服务集团 yangyi01 at inspur.com
Tue Mar 17 09:08:29 UTC 2020


Hi, William

Finally, my highend server is available and so I can do performance comparison again, tpacket_v3 obviously has big performance improvement, here is my data. By the way, in order to get stable performance data, please use taskset to pin ovs-vswitchd to a physical core (you shouldn't schedule other task to its logical sibling core for stable performance data), iperf3 client an iperf3 use different cores, for my case, ovs-vswitchd is pinned to core 1, iperf3 server is pinned to core 4, iperf3 client is pinned to core 5. 

According to my test, tpacket_v3 can get about 55% improvement (from 1.34 to 2.08,  (2.08-1.34)/1.34 = 0.55) , with my further optimization (use zero copy for receive side), it can have more improvement (from 1.34 to 2.21, (2.21-1.34)/1.34 = 0.65), so I still think performance improvement is big, please reconsider it again.

William, I can help you do performance check on your servers if you'd like, from these data and previous data, we can draw a conclusion performance data is very platform sensitive. You can schedule a meeting for further discussion if needed.

No zero copy and no tpacket_v3 (recvmmsg, sendmmsg)
===================================================
eipadmin at eip01:~$ sudo ./run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 43194 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  1.58 GBytes  1.35 Gbits/sec  13851    103 KBytes
[  4]  10.00-20.00  sec  1.56 GBytes  1.34 Gbits/sec  14018   94.7 KBytes
[  4]  20.00-30.00  sec  1.56 GBytes  1.34 Gbits/sec  13942   94.7 KBytes
[  4]  30.00-40.00  sec  1.56 GBytes  1.34 Gbits/sec  13565    106 KBytes
[  4]  40.00-50.00  sec  1.54 GBytes  1.32 Gbits/sec  14567    106 KBytes
[  4]  50.00-60.00  sec  1.56 GBytes  1.34 Gbits/sec  13738   84.8 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  9.35 GBytes  1.34 Gbits/sec  83681             sender
[  4]   0.00-60.00  sec  9.35 GBytes  1.34 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 43192
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43194
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  1.57 GBytes  1.35 Gbits/sec
[  5]  10.00-20.00  sec  1.56 GBytes  1.34 Gbits/sec
[  5]  20.00-30.00  sec  1.56 GBytes  1.34 Gbits/sec
[  5]  30.00-40.00  sec  1.56 GBytes  1.34 Gbits/sec
[  5]  40.00-50.00  sec  1.54 GBytes  1.32 Gbits/sec
[  5]  50.00-60.00  sec  1.56 GBytes  1.34 Gbits/sec


iperf Done.
eipadmin at eip01:~$

No zero copy but with tpacket_v3
================================
eipadmin at eip01:~$ sudo ./run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 43174 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  2.36 GBytes  2.02 Gbits/sec    0   3.04 MBytes
[  4]  10.00-20.00  sec  2.43 GBytes  2.09 Gbits/sec    0   3.04 MBytes
[  4]  20.00-30.00  sec  2.44 GBytes  2.09 Gbits/sec    0   3.04 MBytes
[  4]  30.00-40.00  sec  2.43 GBytes  2.09 Gbits/sec    0   3.04 MBytes
[  4]  40.00-50.00  sec  2.43 GBytes  2.09 Gbits/sec    0   3.04 MBytes
[  4]  50.00-60.00  sec  2.44 GBytes  2.10 Gbits/sec    0   3.04 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  14.5 GBytes  2.08 Gbits/sec    0             sender
[  4]   0.00-60.00  sec  14.5 GBytes  2.08 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 43172
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43174
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  2.35 GBytes  2.02 Gbits/sec
[  5]  10.00-20.00  sec  2.43 GBytes  2.09 Gbits/sec
[  5]  20.00-30.00  sec  2.44 GBytes  2.09 Gbits/sec
[  5]  30.00-40.00  sec  2.43 GBytes  2.09 Gbits/sec
[  5]  40.00-50.00  sec  2.43 GBytes  2.09 Gbits/sec
[  5]  50.00-60.00  sec  2.44 GBytes  2.10 Gbits/sec


iperf Done.
eipadmin at eip01:~$


Have zero copy patch and tpacket_v3
===================================
eipadmin at eip01:~$ sudo ./run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 43182 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  2.54 GBytes  2.18 Gbits/sec    0   3.03 MBytes
[  4]  10.00-20.00  sec  2.58 GBytes  2.22 Gbits/sec    0   3.03 MBytes
[  4]  20.00-30.00  sec  2.58 GBytes  2.22 Gbits/sec    0   3.03 MBytes
[  4]  30.00-40.00  sec  2.59 GBytes  2.22 Gbits/sec    0   3.03 MBytes
[  4]  40.00-50.00  sec  2.57 GBytes  2.21 Gbits/sec    0   3.03 MBytes
[  4]  50.00-60.00  sec  2.57 GBytes  2.21 Gbits/sec    0   3.03 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  15.4 GBytes  2.21 Gbits/sec    0             sender
[  4]   0.00-60.00  sec  15.4 GBytes  2.21 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 43180
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 43182
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  2.53 GBytes  2.17 Gbits/sec
[  5]  10.00-20.00  sec  2.58 GBytes  2.22 Gbits/sec
[  5]  20.00-30.00  sec  2.58 GBytes  2.22 Gbits/sec
[  5]  30.00-40.00  sec  2.59 GBytes  2.22 Gbits/sec
[  5]  40.00-50.00  sec  2.57 GBytes  2.21 Gbits/sec
[  5]  50.00-60.00  sec  2.57 GBytes  2.21 Gbits/sec


iperf Done.
eipadmin at eip01:~$

-----邮件原件-----
发件人: William Tu [mailto:u9012063 at gmail.com] 
发送时间: 2020年3月14日 22:18
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com>
抄送: i.maximets at ovn.org; blp at ovn.org; yang_y_yi at 163.com; ovs-dev at openvswitch.org
主题: Re: [ovs-dev] 答复: [PATCH v6] Use TPACKET_V3 to accelerate veth for userspace datapath

On Fri, Mar 13, 2020 at 9:45 PM Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com> wrote:
>
> Io_uring is a feature brought in by Linux kernel 5.1, so it can't be 
> used on Linux system with kernel version < 5.1. tpacket_v3 is only one 
> way to avoid system call on almost all the Linux kernel versions, it 
> is unique from this perspective. Maybe you will miss it if someone 
> fixes kernel side issue :-)
>
> In addition, according to what Flavio said, TSO can't support VXLAN currently, but in most cloud scenarios, VXLAN is only one choice, so for such cases, TSO can be ignored.
>
> My point is we can provide one option for such use cases, once kernel side issue is fixed, all the Linux distributions can apply this fix, users can get immediate benefits without change. So maybe adding a switch userspace-use-tpacket-v3 in other-config (set to False by default) is an acceptable way to handle this.
>

The tpacket_v3 patch now shows very little performance improvement.
So there is little incentive to merge and maintain this code.
Do you know if kernel side is fixed, will tpacket_v3 have better performance improvement?

Or another way is to study io_uring and compare its performance with tpacket_v3.

William


More information about the dev mailing list