[ovs-discuss] the network performence is not normal when use openvswitch.ko make from ovs tree

Tonghao Zhang xiangxia.m.yue at gmail.com
Tue Nov 5 07:59:06 UTC 2019


On Mon, Nov 4, 2019 at 5:14 PM shuangyang qian <qsyqian at gmail.com> wrote:
>
> Hi:
> I make rpm packages for ovs and ovn with this document:http://docs.openvswitch.org/en/latest/intro/install/fedora/ . For use the kernel module in ovs tree, i configure with the command: ./configure --with-linux=/lib/modules/$(uname -r)/build .
> Then install the rpm packages.
> when i finished, i check the openvswitch.ko is like:
> # lsmod |  grep openvswitch
> openvswitch           291276  0
> tunnel6                 3115  1 openvswitch
> nf_defrag_ipv6         25957  2 nf_conntrack_ipv6,openvswitch
> nf_nat_ipv6             6459  2 openvswitch,ip6table_nat
> nf_nat_ipv4             6187  2 openvswitch,iptable_nat
> nf_nat                 18080  5 xt_nat,openvswitch,nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4
> nf_conntrack          102766  10 ip_vs,nf_conntrack_ipv6,openvswitch,nf_conntrack_ipv4,nf_conntrack_netlink,nf_nat_ipv6,nf_nat_masquerade_ipv4,xt_conntrack,nf_nat_ipv4,nf_nat
> libcrc32c               1388  3 ip_vs,openvswitch,xfs
> ipv6                  400397  92 ip_vs,nf_conntrack_ipv6,openvswitch,nf_defrag_ipv6,nf_nat_ipv6,bridge
> # modinfo openvswitch
> filename:       /lib/modules/4.9.18-19080201/extra/openvswitch/openvswitch.ko
> alias:          net-pf-16-proto-16-family-ovs_ct_limit
> alias:          net-pf-16-proto-16-family-ovs_meter
> alias:          net-pf-16-proto-16-family-ovs_packet
> alias:          net-pf-16-proto-16-family-ovs_flow
> alias:          net-pf-16-proto-16-family-ovs_vport
> alias:          net-pf-16-proto-16-family-ovs_datapath
> version:        2.11.2
> license:        GPL
> description:    Open vSwitch switching datapath
> srcversion:     9DDA327F9DD46B9813628A4
> depends:        nf_conntrack,tunnel6,ipv6,nf_nat,nf_defrag_ipv6,libcrc32c,nf_nat_ipv6,nf_nat_ipv4
> vermagic:       4.9.18-19080201 SMP mod_unload modversions
> parm:           udp_port:Destination UDP port (ushort)
> # rpm -qf /lib/modules/4.9.18-19080201/extra/openvswitch/openvswitch.ko
> openvswitch-kmod-2.11.2-1.el7.x86_64
>
> Then i start to build my network structure. I have two node,and network namespace vm1 on node1, network namespace vm2 on node2. vm1's veth pair veth-vm1 is on node1's br-int. vm2's veth pair veth-vm2 is on node2's br-int. In logical layer, there is one logical switch test-subnet and two logical switch port node1 and node2 on it. like this:
> # ovn-nbctl show
> switch 70585c0e-3cd9-459e-9448-3c13f3c0bfa3 (test-subnet)
>     port node2
>         addresses: ["00:00:00:00:00:02 192.168.100.20"]
>     port node1
>         addresses: ["00:00:00:00:00:01 192.168.100.10"]
> on node1:
> # ovs-vsctl show
> 5180f74a-1379-49af-b265-4403bd0d82d8
>     Bridge br-int
>         fail_mode: secure
>         Port "ovn-431b9e-0"
>             Interface "ovn-431b9e-0"
>                 type: geneve
>                 options: {csum="true", key=flow, remote_ip="10.18.124.2"}
>         Port br-int
>             Interface br-int
>                 type: internal
>         Port "veth-vm1"
>             Interface "veth-vm1"
>     ovs_version: "2.11.2"
> # ip netns exec vm1 ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>        valid_lft forever preferred_lft forever
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
> 14: ovs-gretap0 at NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 15: erspan0 at NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
>     link/ether 22:02:1b:08:ec:53 brd ff:ff:ff:ff:ff:ff
> 16: ovs-ip6gre0 at NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default qlen 1
>     link/gre6 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
> 17: ovs-ip6tnl0 at NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1
>     link/tunnel6 :: brd ::
> 18: vm1-eth0 at if17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
>     link/ether 00:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0
>     inet 192.168.100.10/24 scope global vm1-eth0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::200:ff:fe00:1/64 scope link
>        valid_lft forever preferred_lft forever
>
>
> on node2:# ovs-vsctl show
> 011332d0-78bc-47f7-be3c-fab0beb08e28
>     Bridge br-int
>         fail_mode: secure
>         Port br-int
>             Interface br-int
>                 type: internal
>         Port "ovn-c655f8-0"
>             Interface "ovn-c655f8-0"
>                 type: geneve
>                 options: {csum="true", key=flow, remote_ip="10.18.124.1"}
>         Port "veth-vm2"
>             Interface "veth-vm2"
>     ovs_version: "2.11.2"
> #ip netns exec vm2 ip a
> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
>     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
>     inet 127.0.0.1/8 scope host lo
>        valid_lft forever preferred_lft forever
>     inet6 ::1/128 scope host
>        valid_lft forever preferred_lft forever
> 10: ovs-gretap0 at NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN group default qlen 1000
>     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
> 11: erspan0 at NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN group default qlen 1000
>     link/ether 4a:1d:ca:65:e3:ca brd ff:ff:ff:ff:ff:ff
> 12: ovs-ip6gre0 at NONE: <NOARP> mtu 1448 qdisc noop state DOWN group default qlen 1
>     link/gre6 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00 brd 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00
> 13: ovs-ip6tnl0 at NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1
>     link/tunnel6 :: brd ::
> 17: vm2-eth0 at if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default qlen 1000
>     link/ether 00:00:00:00:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
>     inet 192.168.100.20/24 scope global vm2-eth0
>        valid_lft forever preferred_lft forever
>     inet6 fe80::200:ff:fe00:2/64 scope link
>        valid_lft forever preferred_lft forever
>
> then i start to use iperf to check the network performence, oh, by the way, i use geneve protocol between the two nodes, ovn-sbctl show is :
> # ovn-sbctl show
> Chassis "c655f877-b7ed-4bb5-a047-23521426d541"
>     hostname: "node1.com"
>     Encap geneve
>         ip: "10.18.124.1"
>         options: {csum="true"}
>     Port_Binding "node1"
> Chassis "431b9efb-b464-42a1-a6dd-7fc6e0176137"
>     hostname: "node2.com"
>     Encap geneve
>         ip: "10.18.124.2"
>         options: {csum="true"}
>     Port_Binding "node2"
>
> on node1, in network namespace vm1 i start the iperf3 as server:
> #ip netns exec vm1 iperf3 -s
> on node2, in network namespace vm2 i start the iper3 as client:
> # ip netns exec vm2 iperf3 -c 192.168.100.10
> Connecting to host 192.168.100.10, port 5201
> [  4] local 192.168.100.20 port 40708 connected to 192.168.100.10 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec   431 MBytes  3.61 Gbits/sec   34    253 KBytes
> [  4]   1.00-2.00   sec   426 MBytes  3.58 Gbits/sec    0    253 KBytes
> [  4]   2.00-3.00   sec   426 MBytes  3.57 Gbits/sec    0    253 KBytes
> [  4]   3.00-4.00   sec   401 MBytes  3.37 Gbits/sec    0    255 KBytes
> [  4]   4.00-5.00   sec   429 MBytes  3.60 Gbits/sec    0    255 KBytes
> [  4]   5.00-6.00   sec   413 MBytes  3.46 Gbits/sec    0    253 KBytes
> [  4]   6.00-7.00   sec   409 MBytes  3.43 Gbits/sec    0    250 KBytes
> [  4]   7.00-8.00   sec   427 MBytes  3.58 Gbits/sec    0    253 KBytes
> [  4]   8.00-9.00   sec   417 MBytes  3.49 Gbits/sec    0    250 KBytes
> [  4]   9.00-10.00  sec   385 MBytes  3.23 Gbits/sec    0   5.27 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-10.00  sec  4.07 GBytes  3.49 Gbits/sec   34             sender
> [  4]   0.00-10.00  sec  4.07 GBytes  3.49 Gbits/sec                  receiver
>
> as you see, the bw is only 3.xxGbits/sec, but my physics eth1's bw 10000M:
Hi, I run the ovs on node1 and node2, using geneve tunnel but I didn't
reproduce your issue.

create the geneve tunnel:
# ovs-vsctl add-br br-int
# ovs-vsctl add-port br-int gnv0 -- set Interface gnv0 type=geneve
options:csum=true options:key=100 options:remote_ip=1.1.1.200

# ovs-vsctl show
9393485c-c64c-490e-884e-418ff5d90251
    Bridge br-int
        Port gnv0
            Interface gnv0
                type: geneve
                options: {csum="true", key="100", remote_ip="1.1.1.200"}
        Port __tap01
            Interface __tap01
        Port br-int
            Interface br-int
                type: internal

# ip netns exec ns100 ifconfig
__tap00: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 2.2.2.100  netmask 255.255.255.0  broadcast 0.0.0.0
        inet6 fe80::254:ff:fe00:1  prefixlen 64  scopeid 0x20<link>
        ether 00:54:00:00:00:01  txqueuelen 1000  (Ethernet)
        RX packets 605000  bytes 39951500 (38.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 819864  bytes 31247862764 (29.1 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

# ip netns exec ns100 iperf -c 2.2.2.200 -i 2 -t 10
------------------------------------------------------------
Client connecting to 2.2.2.200, TCP port 5001
TCP window size:  482 KByte (default)
------------------------------------------------------------
[  3] local 2.2.2.100 port 41428 connected with 2.2.2.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  1.85 GBytes  7.93 Gbits/sec
[  3]  2.0- 4.0 sec  1.94 GBytes  8.33 Gbits/sec

# modinfo openvswitch
filename:       /lib/modules/3.10.0-957.1.3.el7.x86_64/extra/openvswitch.ko

so, can you use the commands as show above to reproduce your
issue.(the kernel version is different.)
> # ethtool eth1
> Settings for eth1:
>         Supported ports: [ FIBRE ]
>         Supported link modes:   10000baseT/Full
>         Supported pause frame use: Symmetric
>         Supports auto-negotiation: No
>         Supported FEC modes: Not reported
>         Advertised link modes:  10000baseT/Full
>         Advertised pause frame use: Symmetric
>         Advertised auto-negotiation: No
>         Advertised FEC modes: Not reported
>         Speed: 10000Mb/s
>         Duplex: Full
>         Port: Other
>         PHYAD: 0
>         Transceiver: external
>         Auto-negotiation: off
>         Supports Wake-on: d
>         Wake-on: d
>         Current message level: 0x00000007 (7)
>                                drv probe link
>         Link detected: yes
>
> when i uninstall the openvswitch-kmod package and use the openvswitch.ko in the upstream linux kernel, like this:
> # lsmod | grep openvswitch
> openvswitch            95805  0
> nf_defrag_ipv6         25957  2 nf_conntrack_ipv6,openvswitch
> nf_nat_ipv6             6459  2 openvswitch,ip6table_nat
> nf_nat_ipv4             6187  2 openvswitch,iptable_nat
> nf_nat                 18080  5 xt_nat,openvswitch,nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4
> nf_conntrack          102766  10 ip_vs,nf_conntrack_ipv6,openvswitch,nf_conntrack_ipv4,nf_conntrack_netlink,nf_nat_ipv6,nf_nat_masquerade_ipv4,xt_conntrack,nf_nat_ipv4,nf_nat
> libcrc32c               1388  3 ip_vs,openvswitch,xfs
> # modinfo openvswitch
> filename:       /lib/modules/4.9.18-19080201/kernel/net/openvswitch/openvswitch.ko
> alias:          net-pf-16-proto-16-family-ovs_packet
> alias:          net-pf-16-proto-16-family-ovs_flow
> alias:          net-pf-16-proto-16-family-ovs_vport
> alias:          net-pf-16-proto-16-family-ovs_datapath
> license:        GPL
> description:    Open vSwitch switching datapath
> srcversion:     915B872C96FB1D38D107742
> depends:        nf_conntrack,nf_nat,libcrc32c,nf_nat_ipv6,nf_nat_ipv4,nf_defrag_ipv6
> intree:         Y
> vermagic:       4.9.18-19080201 SMP mod_unload modversions
>
> and do the same test in above, and i get the follow result:
> # ip netns exec vm2 iperf3 -c 192.168.100.10
> Connecting to host 192.168.100.10, port 5201
> [  4] local 192.168.100.20 port 40652 connected to 192.168.100.10 port 5201
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec  1000 MBytes  8.39 Gbits/sec    4    290 KBytes
> [  4]   1.00-2.00   sec   994 MBytes  8.34 Gbits/sec    0    292 KBytes
> [  4]   2.00-3.00   sec  1002 MBytes  8.41 Gbits/sec    0    287 KBytes
>    4]   3.00-4.00   sec   994 MBytes  8.34 Gbits/sec    0    292 KBytes
> ▽  4]   4.00-5.00   sec   992 MBytes  8.32 Gbits/sec    0    298 KBytes
> [  4]   5.00-6.00   sec   994 MBytes  8.34 Gbits/sec    0    305 KBytes
> [  4]   6.00-7.00   sec   989 MBytes  8.29 Gbits/sec    0    313 KBytes
> [  4]   7.00-8.00   sec   992 MBytes  8.32 Gbits/sec    0    290 KBytes
> [  4]   8.00-9.00   sec   996 MBytes  8.36 Gbits/sec    0    303 KBytes
> [  4]   9.00-10.00  sec   955 MBytes  8.01 Gbits/sec    0   5.27 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval           Transfer     Bandwidth       Retr
> [  4]   0.00-10.00  sec  9.67 GBytes  8.31 Gbits/sec    4             sender
> [  4]   0.00-10.00  sec  9.67 GBytes  8.31 Gbits/sec                  receiver
>
> so, i cann't understand why the performence is so poor when i use the kernel module build from the ovs tree.
>
> anyone can give me some advice where is wrong?
>
> thx!
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss


More information about the discuss mailing list