[ovs-discuss] poor ovs-dpdk performance vs ovs

Lam, Tiago tiago.lam at intel.com
Fri Nov 16 10:06:39 UTC 2018


Hi,

I have a couple of questions, but at a first glance this seems to be a
result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
in the works of addressing that.

The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
to the ones we have been getting with and without our TSO series [1]
applied.

On the description below, is this a virtualized environment by any
change? I.e. are you running the iperf tests on top of VMs running on
those Centos hosts, or directly on them?

Can you also confirm you are using TSO when using OvS with the Kernel by
reporting what you get with $ethtool -k $iface | grep
"tcp-segmentation-offload:". Is it reported as "on"?

Regards,
Tiago.

[1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html

On 15/11/2018 22:17, Robert Brooks wrote:
> I have a pair of Centos 7 hosts on 10gig switch.
> 
> Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf test.
> 
> After switching the receiving host to ovs-dpdk following guidance here:
> http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> 
> I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
> 
> Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB
> RAM, Intel 92599ES NIC.
> 
> I have confirmed the doucmented kernel boot options are set and 1GB
> hugepages are in use.
> 
> # cat /proc/cmdline 
> BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
> root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
> rd.lvm.lv <http://rd.lvm.lv>=vg01/swap rd.lvm.lv
> <http://rd.lvm.lv>=vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on
> default_hugepagesz=1G hugepagesz=1G hugepages=8
> 
> Software is openvswitch 2.10.1 built using the provided rpm spec file. I
> have used both the Centos provided dpdk 17.11 and a re-compile using
> latest LTS of 17.11.4.
> 
> I have tried various performance tweaks, including cpu pinning and
> isolated cpus.
> 
> Switching back to regular ovs returns the performance to close to wire
> speed.
> 
> Under load the following is logged:
> 
> 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on
> fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
> CPU usage)
> 
> Config dump:
> 
> # ovsdb-client dump Open_vSwitch
> 
> AutoAttach table
> 
> _uuid mappings system_description system_name
> 
> ----- -------- ------------------ -----------
> 
> 
> Bridge table
> 
> _uuid                                auto_attach controller datapath_id 
>       datapath_type datapath_version external_ids fail_mode flood_vlans
> flow_tables ipfix mcast_snooping_enable mirrors name  netflow
> other_config ports                                                      
>          
> 
> ------------------------------------ ----------- ----------
> ------------------ ------------- ---------------- ------------ ---------
> ----------- ----------- ----- --------------------- ------- -----
> ------- ------------
> ---------------------------------------------------------------------
> 
> 006f021a-b14d-49fb-a11d-48df2fa2bca1 []          []        
> "0000001b21a6ddc4" netdev        "<built-in>"     {}           []       
> []          {}          []    false                 []      "br0" []   
>   {}           [3bd79dba-777c-40d0-b573-bf9e027326f4,
> 60b0661f-2177-4283-a6cb-a80336
> 
> 
> Controller table
> 
> _uuid connection_mode controller_burst_limit controller_rate_limit
> enable_async_messages external_ids inactivity_probe is_connected
> local_gateway local_ip local_netmask max_backoff other_config role
> status target
> 
> ----- --------------- ---------------------- ---------------------
> --------------------- ------------ ---------------- ------------
> ------------- -------- ------------- ----------- ------------ ----
> ------ ------
> 
> 
> Flow_Sample_Collector_Set table
> 
> _uuid bridge external_ids id ipfix
> 
> ----- ------ ------------ -- -----
> 
> 
> Flow_Table table
> 
> _uuid external_ids flow_limit groups name overflow_policy prefixes
> 
> ----- ------------ ---------- ------ ---- --------------- --------
> 
> 
> IPFIX table
> 
> _uuid cache_active_timeout cache_max_flows external_ids obs_domain_id
> obs_point_id other_config sampling targets
> 
> ----- -------------------- --------------- ------------ -------------
> ------------ ------------ -------- -------
> 
> 
> Interface table
> 
> _uuid                                admin_state bfd bfd_status
> cfm_fault cfm_fault_status cfm_flap_count cfm_health cfm_mpid
> cfm_remote_mpids cfm_remote_opstate duplex error external_ids ifindex 
> ingress_policing_burst ingress_policing_rate lacp_current link_resets
> link_speed  link_stat 
> 
> ------------------------------------ ----------- --- ----------
> --------- ---------------- -------------- ---------- --------
> ---------------- ------------------ ------ ----- ------------ --------
> ---------------------- --------------------- ------------ -----------
> ----------- ----------
> 
> 271077eb-97af-4c2d-a5ee-2c63c9367312 up          {}  {}         []     
>   []               []             []         []       []              
> []                 full   []    {}           13       0                 
>     0                     []           11          10000000    up        
> 
> 4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136 up          {}  {}         []     
>   []               []             []         []       []              
> []                 full   []    {}           13032858 0                 
>     0                     []           0           10000000000 up       s
> 
> 
> Manager table
> 
> _uuid connection_mode external_ids inactivity_probe is_connected
> max_backoff other_config status target
> 
> ----- --------------- ------------ ---------------- ------------
> ----------- ------------ ------ ------
> 
> 
> Mirror table
> 
> _uuid external_ids name output_port output_vlan select_all
> select_dst_port select_src_port select_vlan snaplen statistics
> 
> ----- ------------ ---- ----------- ----------- ----------
> --------------- --------------- ----------- ------- ----------
> 
> 
> NetFlow table
> 
> _uuid active_timeout add_id_to_interface engine_id engine_type
> external_ids targets
> 
> ----- -------------- ------------------- --------- -----------
> ------------ -------
> 
> 
> Open_vSwitch table
> 
> _uuid                                bridges                           
>     cur_cfg datapath_types   db_version dpdk_initialized dpdk_version  
> external_ids                                                            
>                                 iface_types                             
> 
> ------------------------------------
> -------------------------------------- ------- ----------------
> ---------- ---------------- --------------
> --------------------------------------------------------------------------------------------------------
> ----------------------------------------
> 
> ea8e7bcc-df11-42bd-96c9-b7eaafdfa863
> [006f021a-b14d-49fb-a11d-48df2fa2bca1] 45      [netdev, system] "7.16.1"
>   true             "DPDK 17.11.4" {hostname="v41",
> rundir="/var/run/openvswitch",
> system-id="7d5d1db0-87db-41bc-bb94-661dde831e9f"} [dpdk, dpdkr,
> dpdkvhostuser, dpdkvhostum
> 
> 
> Port table
> 
> _uuid                                bond_active_slave bond_downdelay
> bond_fake_iface bond_mode bond_updelay cvlans external_ids fake_bridge
> interfaces                             lacp mac name   other_config
> protected qos rstp_statistics rstp_status statistics status tag trunks
> vlan_mode
> 
> ------------------------------------ ----------------- --------------
> --------------- --------- ------------ ------ ------------ -----------
> -------------------------------------- ---- --- ------ ------------
> --------- --- --------------- ----------- ---------- ------ --- ------
> ---------
> 
> 60b0661f-2177-4283-a6cb-a80336a8c478 []                0             
> false           []        0            []     {}           false      
> [271077eb-97af-4c2d-a5ee-2c63c9367312] []   []  "br0"  {}          
> false     []  {}              {}          {}         {}     []  []     []
> 
> 3bd79dba-777c-40d0-b573-bf9e027326f4 []                0             
> false           []        0            []     {}           false      
> [4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136] []   []  "p2p1" {}          
> false     []  {}              {}          {}         {}     []  []     []
> 
> 
> QoS table
> 
> _uuid external_ids other_config queues type
> 
> ----- ------------ ------------ ------ ----
> 
> 
> Queue table
> 
> _uuid dscp external_ids other_config
> 
> ----- ---- ------------ ------------
> 
> 
> SSL table
> 
> _uuid bootstrap_ca_cert ca_cert certificate external_ids private_key
> 
> ----- ----------------- ------- ----------- ------------ -----------
> 
> 
> sFlow table
> 
> _uuid agent external_ids header polling sampling targets
> 
> ----- ----- ------------ ------ ------- -------- -------
> 
> 
> I see rx_dropped counters. Other than that I'm not sure what's going on,
> as I've tried various tuning suggested by dpdk, Intel and Red Hat, but
> performance remains unimproved.
> 
> 
> Regards,
> 
> 
> Rob
> 
> 
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> 


More information about the discuss mailing list