[ovs-discuss] poor ovs-dpdk performance vs ovs
Robert Brooks
robmbrooks at gmail.com
Fri Nov 16 18:52:22 UTC 2018
Hi Tiago,
you are correct, this is a virtualization environment. The tests numbers I
am quoting are running directly on the hypervisor, rather than guests.
My ultimate aim is to get good performance for guests talking over VXLAN.
Using OvS I get around 5Gb/s between guests talking over VXLAN and near
wire speed performance where the guests talk to a bridge without VXLAN. I'd
like to improve on the VXLAN performance.
I believe VXLAN is offloaded by tx-udp_tnl-segmentation? (...VXLAN is being
configured at the interface level by OpenNebula, so handled by the kernel
module).
Regarding the NIC, I see:
[root at v41 ~]# ethtool -k p2p1 | grep tcp-segmentation-offload
*tcp-segmentation-offload*: on
Are the TSO patches for OvS-DPDK suitable to apply and use, if so anything
I should know? eg version to apply to, source to obtain current patches
from?
Many thanks,
Rob
On Fri, Nov 16, 2018 at 2:06 AM Lam, Tiago <tiago.lam at intel.com> wrote:
> Hi,
>
> I have a couple of questions, but at a first glance this seems to be a
> result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
> in the works of addressing that.
>
> The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
> to the ones we have been getting with and without our TSO series [1]
> applied.
>
> On the description below, is this a virtualized environment by any
> change? I.e. are you running the iperf tests on top of VMs running on
> those Centos hosts, or directly on them?
>
> Can you also confirm you are using TSO when using OvS with the Kernel by
> reporting what you get with $ethtool -k $iface | grep
> "tcp-segmentation-offload:". Is it reported as "on"?
>
> Regards,
> Tiago.
>
> [1] https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html
>
> On 15/11/2018 22:17, Robert Brooks wrote:
> > I have a pair of Centos 7 hosts on 10gig switch.
> >
> > Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf
> test.
> >
> > After switching the receiving host to ovs-dpdk following guidance here:
> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> >
> > I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
> >
> > Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB
> > RAM, Intel 92599ES NIC.
> >
> > I have confirmed the doucmented kernel boot options are set and 1GB
> > hugepages are in use.
> >
> > # cat /proc/cmdline
> > BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
> > root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
> > rd.lvm.lv <http://rd.lvm.lv>=vg01/swap rd.lvm.lv
> > <http://rd.lvm.lv>=vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on
> > default_hugepagesz=1G hugepagesz=1G hugepages=8
> >
> > Software is openvswitch 2.10.1 built using the provided rpm spec file. I
> > have used both the Centos provided dpdk 17.11 and a re-compile using
> > latest LTS of 17.11.4.
> >
> > I have tried various performance tweaks, including cpu pinning and
> > isolated cpus.
> >
> > Switching back to regular ovs returns the performance to close to wire
> > speed.
> >
> > Under load the following is logged:
> >
> > 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on
> > fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
> > CPU usage)
> >
> > Config dump:
> >
> > # ovsdb-client dump Open_vSwitch
> >
> > AutoAttach table
> >
> > _uuid mappings system_description system_name
> >
> > ----- -------- ------------------ -----------
> >
> >
> > Bridge table
> >
> > _uuid auto_attach controller datapath_id
> > datapath_type datapath_version external_ids fail_mode flood_vlans
> > flow_tables ipfix mcast_snooping_enable mirrors name netflow
> > other_config ports
> >
> >
> > ------------------------------------ ----------- ----------
> > ------------------ ------------- ---------------- ------------ ---------
> > ----------- ----------- ----- --------------------- ------- -----
> > ------- ------------
> > ---------------------------------------------------------------------
> >
> > 006f021a-b14d-49fb-a11d-48df2fa2bca1 [] []
> > "0000001b21a6ddc4" netdev "<built-in>" {} []
> > [] {} [] false [] "br0" []
> > {} [3bd79dba-777c-40d0-b573-bf9e027326f4,
> > 60b0661f-2177-4283-a6cb-a80336
> >
> >
> > Controller table
> >
> > _uuid connection_mode controller_burst_limit controller_rate_limit
> > enable_async_messages external_ids inactivity_probe is_connected
> > local_gateway local_ip local_netmask max_backoff other_config role
> > status target
> >
> > ----- --------------- ---------------------- ---------------------
> > --------------------- ------------ ---------------- ------------
> > ------------- -------- ------------- ----------- ------------ ----
> > ------ ------
> >
> >
> > Flow_Sample_Collector_Set table
> >
> > _uuid bridge external_ids id ipfix
> >
> > ----- ------ ------------ -- -----
> >
> >
> > Flow_Table table
> >
> > _uuid external_ids flow_limit groups name overflow_policy prefixes
> >
> > ----- ------------ ---------- ------ ---- --------------- --------
> >
> >
> > IPFIX table
> >
> > _uuid cache_active_timeout cache_max_flows external_ids obs_domain_id
> > obs_point_id other_config sampling targets
> >
> > ----- -------------------- --------------- ------------ -------------
> > ------------ ------------ -------- -------
> >
> >
> > Interface table
> >
> > _uuid admin_state bfd bfd_status
> > cfm_fault cfm_fault_status cfm_flap_count cfm_health cfm_mpid
> > cfm_remote_mpids cfm_remote_opstate duplex error external_ids ifindex
> > ingress_policing_burst ingress_policing_rate lacp_current link_resets
> > link_speed link_stat
> >
> > ------------------------------------ ----------- --- ----------
> > --------- ---------------- -------------- ---------- --------
> > ---------------- ------------------ ------ ----- ------------ --------
> > ---------------------- --------------------- ------------ -----------
> > ----------- ----------
> >
> > 271077eb-97af-4c2d-a5ee-2c63c9367312 up {} {} []
> > [] [] [] [] []
> > [] full [] {} 13 0
> > 0 [] 11 10000000 up
> >
> > 4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136 up {} {} []
> > [] [] [] [] []
> > [] full [] {} 13032858 0
> > 0 [] 0 10000000000 up s
> >
> >
> > Manager table
> >
> > _uuid connection_mode external_ids inactivity_probe is_connected
> > max_backoff other_config status target
> >
> > ----- --------------- ------------ ---------------- ------------
> > ----------- ------------ ------ ------
> >
> >
> > Mirror table
> >
> > _uuid external_ids name output_port output_vlan select_all
> > select_dst_port select_src_port select_vlan snaplen statistics
> >
> > ----- ------------ ---- ----------- ----------- ----------
> > --------------- --------------- ----------- ------- ----------
> >
> >
> > NetFlow table
> >
> > _uuid active_timeout add_id_to_interface engine_id engine_type
> > external_ids targets
> >
> > ----- -------------- ------------------- --------- -----------
> > ------------ -------
> >
> >
> > Open_vSwitch table
> >
> > _uuid bridges
> > cur_cfg datapath_types db_version dpdk_initialized dpdk_version
> > external_ids
> > iface_types
> >
> > ------------------------------------
> > -------------------------------------- ------- ----------------
> > ---------- ---------------- --------------
> >
> --------------------------------------------------------------------------------------------------------
> > ----------------------------------------
> >
> > ea8e7bcc-df11-42bd-96c9-b7eaafdfa863
> > [006f021a-b14d-49fb-a11d-48df2fa2bca1] 45 [netdev, system] "7.16.1"
> > true "DPDK 17.11.4" {hostname="v41",
> > rundir="/var/run/openvswitch",
> > system-id="7d5d1db0-87db-41bc-bb94-661dde831e9f"} [dpdk, dpdkr,
> > dpdkvhostuser, dpdkvhostum
> >
> >
> > Port table
> >
> > _uuid bond_active_slave bond_downdelay
> > bond_fake_iface bond_mode bond_updelay cvlans external_ids fake_bridge
> > interfaces lacp mac name other_config
> > protected qos rstp_statistics rstp_status statistics status tag trunks
> > vlan_mode
> >
> > ------------------------------------ ----------------- --------------
> > --------------- --------- ------------ ------ ------------ -----------
> > -------------------------------------- ---- --- ------ ------------
> > --------- --- --------------- ----------- ---------- ------ --- ------
> > ---------
> >
> > 60b0661f-2177-4283-a6cb-a80336a8c478 [] 0
> > false [] 0 [] {} false
> > [271077eb-97af-4c2d-a5ee-2c63c9367312] [] [] "br0" {}
> > false [] {} {} {} {} [] [] []
> >
> > 3bd79dba-777c-40d0-b573-bf9e027326f4 [] 0
> > false [] 0 [] {} false
> > [4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136] [] [] "p2p1" {}
> > false [] {} {} {} {} [] [] []
> >
> >
> > QoS table
> >
> > _uuid external_ids other_config queues type
> >
> > ----- ------------ ------------ ------ ----
> >
> >
> > Queue table
> >
> > _uuid dscp external_ids other_config
> >
> > ----- ---- ------------ ------------
> >
> >
> > SSL table
> >
> > _uuid bootstrap_ca_cert ca_cert certificate external_ids private_key
> >
> > ----- ----------------- ------- ----------- ------------ -----------
> >
> >
> > sFlow table
> >
> > _uuid agent external_ids header polling sampling targets
> >
> > ----- ----- ------------ ------ ------- -------- -------
> >
> >
> > I see rx_dropped counters. Other than that I'm not sure what's going on,
> > as I've tried various tuning suggested by dpdk, Intel and Red Hat, but
> > performance remains unimproved.
> >
> >
> > Regards,
> >
> >
> > Rob
> >
> >
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20181116/dcdb4fdf/attachment-0001.html>
More information about the discuss
mailing list