[ovs-dev] 答复: 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath

Yi Yang (杨燚)-云服务集团 yangyi01 at inspur.com
Mon Mar 23 10:00:11 UTC 2020


Hi, folks

I implemented my goal in Ubuntu kernel 4.15.0-92.93, here is my performance data with tpacket_v3 and tso. So now I'm very sure tpacket_v3 can do better.

[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
iperf3: no process found
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 44976 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  19.6 GBytes  16.8 Gbits/sec  106586    307 KBytes
[  4]  10.00-20.00  sec  19.5 GBytes  16.7 Gbits/sec  104625    215 KBytes
[  4]  20.00-30.00  sec  20.0 GBytes  17.2 Gbits/sec  106962    301 KBytes
[  4]  30.00-40.00  sec  19.9 GBytes  17.1 Gbits/sec  102262    346 KBytes
[  4]  40.00-50.00  sec  19.8 GBytes  17.0 Gbits/sec  105383    225 KBytes
[  4]  50.00-60.00  sec  19.9 GBytes  17.1 Gbits/sec  103177    294 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec   119 GBytes  17.0 Gbits/sec  628995             sender
[  4]   0.00-60.00  sec   119 GBytes  17.0 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 44974
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 44976
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  19.5 GBytes  16.7 Gbits/sec
[  5]  10.00-20.00  sec  19.5 GBytes  16.7 Gbits/sec
[  5]  20.00-30.00  sec  20.0 GBytes  17.2 Gbits/sec
[  5]  30.00-40.00  sec  19.9 GBytes  17.1 Gbits/sec
[  5]  40.00-50.00  sec  19.8 GBytes  17.0 Gbits/sec
[  5]  50.00-60.00  sec  19.9 GBytes  17.1 Gbits/sec
[  5]  60.00-60.04  sec  89.1 MBytes  17.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-60.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-60.04  sec   119 GBytes  17.0 Gbits/sec                  receiver


iperf Done.
[yangyi at localhost ovs-master]$

-----邮件原件-----
发件人: Yi Yang (杨燚)-云服务集团 
发送时间: 2020年3月19日 11:12
收件人: 'u9012063 at gmail.com' <u9012063 at gmail.com>
抄送: 'i.maximets at ovn.org' <i.maximets at ovn.org>; 'yang_y_yi at 163.com' <yang_y_yi at 163.com>; 'ovs-dev at openvswitch.org' <ovs-dev at openvswitch.org>
主题: 答复: [ovs-dev] 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath
重要性: 高

Hi, folks

As I said, TPACKET_V3 does have kernel implementation issue, I tried to fix it in Linux kernel 5.5.9, here is my test data with tpacket_v3 and tso enabled. On my low end server, my goal is to reach 16Gbps at least, I still have another idea to improve it.

[yangyi at localhost ovs-master]$ sudo ../run-iperf3.sh
Connecting to host 10.15.1.3, port 5201
[  4] local 10.15.1.2 port 42336 connected to 10.15.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec    1   3.09 MBytes
[  4]  10.00-20.00  sec  12.9 GBytes  11.1 Gbits/sec    0   3.09 MBytes
[  4]  20.00-30.00  sec  12.9 GBytes  11.1 Gbits/sec    3   3.09 MBytes
[  4]  30.00-40.00  sec  12.8 GBytes  11.0 Gbits/sec    0   3.09 MBytes
[  4]  40.00-50.00  sec  12.8 GBytes  11.0 Gbits/sec    0   3.09 MBytes
[  4]  50.00-60.00  sec  12.8 GBytes  11.0 Gbits/sec    0   3.09 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  77.2 GBytes  11.1 Gbits/sec    4             sender
[  4]   0.00-60.00  sec  77.2 GBytes  11.1 Gbits/sec                  receiver

Server output:
Accepted connection from 10.15.1.2, port 42334
[  5] local 10.15.1.3 port 5201 connected to 10.15.1.2 port 42336
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  12.9 GBytes  11.1 Gbits/sec
[  5]  10.00-20.00  sec  12.9 GBytes  11.1 Gbits/sec
[  5]  20.00-30.00  sec  12.9 GBytes  11.1 Gbits/sec
[  5]  30.00-40.00  sec  12.8 GBytes  11.0 Gbits/sec
[  5]  40.00-50.00  sec  12.8 GBytes  11.0 Gbits/sec
[  5]  50.00-60.00  sec  12.8 GBytes  11.0 Gbits/sec
[  5]  60.00-60.01  sec  14.3 MBytes  12.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-60.01  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-60.01  sec  77.2 GBytes  11.0 Gbits/sec                  receiver


iperf Done.
[yangyi at localhost ovs-master]$

-----邮件原件-----
发件人: William Tu [mailto:u9012063 at gmail.com] 
发送时间: 2020年3月19日 5:42
收件人: Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com>
抄送: i.maximets at ovn.org; yang_y_yi at 163.com; ovs-dev at openvswitch.org
主题: Re: [ovs-dev] 答复: [PATCH v7] Use TPACKET_V3 to accelerate veth for userspace datapath

On Wed, Mar 18, 2020 at 6:22 AM Yi Yang (杨燚)-云服务集团 <yangyi01 at inspur.com> wrote:
>
> Ilya, raw socket for the interface type of which is "system" has been 
> set to non-block mode, can you explain which syscall will lead to 
> sleep? Yes, pmd thread will consume CPU resource even if it has 
> nothing to do, but all the type=dpdk ports are handled by pmd thread, 
> here we just let system interfaces look like a DPDK interface. I 
> didn't see any problem in my test, it will be better if you can tell 
> me what will result in a problem and how I can reproduce it. By the 
> way, type=tap/internal interfaces are still be handled by ovs-vswitchd thread.
>
> In addition, only one line change is there, ".is_pmd = true,", 
> ".is_pmd = false," will keep it in ovs-vswitchd if there is any other 
> concern. We can change non-thread-safe parts to support pmd.
>

Hi Yiyang an Ilya,

How about making tpacket_v3 a new netdev class with type="tpacket"?
Like my original patch:
https://mail.openvswitch.org/pipermail/ovs-dev/2019-December/366229.html

Users have to create it specifically by doing type="tpacket", ex:
  $ ovs-vsctl add-port br0 enp2s0 -- set interface enp2s0 type="tpacket"
And we can set is_pmd=true for this particular type.

Regards
William


More information about the dev mailing list