[ovs-discuss] poor ovs-dpdk performance vs ovs
Ian Stokes
ian.stokes at intel.com
Fri Nov 16 22:38:33 UTC 2018
On 11/16/2018 6:52 PM, Robert Brooks wrote:
> Hi Tiago,
>
> you are correct, this is a virtualization environment. The tests numbers
> I am quoting are running directly on the hypervisor, rather than guests.
>
> My ultimate aim is to get good performance for guests talking over
> VXLAN. Using OvS I get around 5Gb/s between guests talking over VXLAN
> and near wire speed performance where the guests talk to a bridge
> without VXLAN. I'd like to improve on the VXLAN performance.
>
> I believe VXLAN is offloaded by tx-udp_tnl-segmentation? (...VXLAN is
> being configured at the interface level by OpenNebula, so handled by the
> kernel module).
>
> Regarding the NIC, I see:
>
> [root at v41 ~]# ethtool -k p2p1 | grep tcp-segmentation-offload
>
> *tcp-segmentation-offload*: on
>
>
> Are the TSO patches for OvS-DPDK suitable to apply and use, if so
> anything I should know? eg version to apply to, source to obtain current
> patches from?
>
Hi Robert,
You can download Tiagos patch set from below:
https://patchwork.ozlabs.org/project/openvswitch/list/?series=70068&state=*
Cover note can be found at
https://mail.openvswitch.org/pipermail/ovs-dev/2018-October/352889.html
Note the first 3 patches of the series have been upstreamed to master
already, the rest may not apply cleanly though at this point.
If you are interested in testing the series you can checkout the
following commit on master that the series was rebased to.
35fe9ef ("dpif-netdev: Add vlan to mask for flow_put operation.")
Thanks
Ian
> Many thanks,
>
> Rob
>
> On Fri, Nov 16, 2018 at 2:06 AM Lam, Tiago <tiago.lam at intel.com
> <mailto:tiago.lam at intel.com>> wrote:
>
> Hi,
>
> I have a couple of questions, but at a first glance this seems to be a
> result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are
> in the works of addressing that.
>
> The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar
> to the ones we have been getting with and without our TSO series [1]
> applied.
>
> On the description below, is this a virtualized environment by any
> change? I.e. are you running the iperf tests on top of VMs running on
> those Centos hosts, or directly on them?
>
> Can you also confirm you are using TSO when using OvS with the Kernel by
> reporting what you get with $ethtool -k $iface | grep
> "tcp-segmentation-offload:". Is it reported as "on"?
>
> Regards,
> Tiago.
>
> [1]
> https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html
>
> On 15/11/2018 22:17, Robert Brooks wrote:
> > I have a pair of Centos 7 hosts on 10gig switch.
> >
> > Using OVS without dpdk enabled I can get 9.4Gb/s with a simple
> iperf test.
> >
> > After switching the receiving host to ovs-dpdk following guidance
> here:
> > http://docs.openvswitch.org/en/latest/intro/install/dpdk/
> >
> > I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.
> >
> > Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus,
> 256GB
> > RAM, Intel 92599ES NIC.
> >
> > I have confirmed the doucmented kernel boot options are set and 1GB
> > hugepages are in use.
> >
> > # cat /proc/cmdline
> > BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64
> > root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto
> > rd.lvm.lv <http://rd.lvm.lv> <http://rd.lvm.lv>=vg01/swap
> rd.lvm.lv <http://rd.lvm.lv>
> > <http://rd.lvm.lv>=vg01/usr console=ttyS0,115200 iommu=pt
> intel_iommu=on
> > default_hugepagesz=1G hugepagesz=1G hugepages=8
> >
> > Software is openvswitch 2.10.1 built using the provided rpm spec
> file. I
> > have used both the Centos provided dpdk 17.11 and a re-compile using
> > latest LTS of 17.11.4.
> >
> > I have tried various performance tweaks, including cpu pinning and
> > isolated cpus.
> >
> > Switching back to regular ovs returns the performance to close to
> wire
> > speed.
> >
> > Under load the following is logged:
> >
> > 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to
> [POLLIN] on
> > fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%
> > CPU usage)
> >
> > Config dump:
> >
> > # ovsdb-client dump Open_vSwitch
> >
> > AutoAttach table
> >
> > _uuid mappings system_description system_name
> >
> > ----- -------- ------------------ -----------
> >
> >
> > Bridge table
> >
> > _uuid auto_attach controller
> datapath_id
> > datapath_type datapath_version external_ids fail_mode
> flood_vlans
> > flow_tables ipfix mcast_snooping_enable mirrors name netflow
> > other_config ports
> >
> >
> > ------------------------------------ ----------- ----------
> > ------------------ ------------- ---------------- ------------
> ---------
> > ----------- ----------- ----- --------------------- ------- -----
> > ------- ------------
> > ---------------------------------------------------------------------
> >
> > 006f021a-b14d-49fb-a11d-48df2fa2bca1 [] []
> > "0000001b21a6ddc4" netdev "<built-in>" {} []
> > [] {} [] false [] "br0" []
> > {} [3bd79dba-777c-40d0-b573-bf9e027326f4,
> > 60b0661f-2177-4283-a6cb-a80336
> >
> >
> > Controller table
> >
> > _uuid connection_mode controller_burst_limit controller_rate_limit
> > enable_async_messages external_ids inactivity_probe is_connected
> > local_gateway local_ip local_netmask max_backoff other_config role
> > status target
> >
> > ----- --------------- ---------------------- ---------------------
> > --------------------- ------------ ---------------- ------------
> > ------------- -------- ------------- ----------- ------------ ----
> > ------ ------
> >
> >
> > Flow_Sample_Collector_Set table
> >
> > _uuid bridge external_ids id ipfix
> >
> > ----- ------ ------------ -- -----
> >
> >
> > Flow_Table table
> >
> > _uuid external_ids flow_limit groups name overflow_policy prefixes
> >
> > ----- ------------ ---------- ------ ---- --------------- --------
> >
> >
> > IPFIX table
> >
> > _uuid cache_active_timeout cache_max_flows external_ids obs_domain_id
> > obs_point_id other_config sampling targets
> >
> > ----- -------------------- --------------- ------------ -------------
> > ------------ ------------ -------- -------
> >
> >
> > Interface table
> >
> > _uuid admin_state bfd bfd_status
> > cfm_fault cfm_fault_status cfm_flap_count cfm_health cfm_mpid
> > cfm_remote_mpids cfm_remote_opstate duplex error external_ids
> ifindex
> > ingress_policing_burst ingress_policing_rate lacp_current link_resets
> > link_speed link_stat
> >
> > ------------------------------------ ----------- --- ----------
> > --------- ---------------- -------------- ---------- --------
> > ---------------- ------------------ ------ ----- ------------
> --------
> > ---------------------- --------------------- ------------ -----------
> > ----------- ----------
> >
> > 271077eb-97af-4c2d-a5ee-2c63c9367312 up {} {} []
> > [] [] [] [] []
> > [] full [] {} 13 0
> > 0 [] 11 10000000 up
> >
> > 4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136 up {} {} []
> > [] [] [] [] []
> > [] full [] {} 13032858 0
> > 0 [] 0 10000000000 up
> s
> >
> >
> > Manager table
> >
> > _uuid connection_mode external_ids inactivity_probe is_connected
> > max_backoff other_config status target
> >
> > ----- --------------- ------------ ---------------- ------------
> > ----------- ------------ ------ ------
> >
> >
> > Mirror table
> >
> > _uuid external_ids name output_port output_vlan select_all
> > select_dst_port select_src_port select_vlan snaplen statistics
> >
> > ----- ------------ ---- ----------- ----------- ----------
> > --------------- --------------- ----------- ------- ----------
> >
> >
> > NetFlow table
> >
> > _uuid active_timeout add_id_to_interface engine_id engine_type
> > external_ids targets
> >
> > ----- -------------- ------------------- --------- -----------
> > ------------ -------
> >
> >
> > Open_vSwitch table
> >
> > _uuid bridges
> > cur_cfg datapath_types db_version dpdk_initialized
> dpdk_version
> > external_ids
> > iface_types
> >
> > ------------------------------------
> > -------------------------------------- ------- ----------------
> > ---------- ---------------- --------------
> >
> --------------------------------------------------------------------------------------------------------
> > ----------------------------------------
> >
> > ea8e7bcc-df11-42bd-96c9-b7eaafdfa863
> > [006f021a-b14d-49fb-a11d-48df2fa2bca1] 45 [netdev, system]
> "7.16.1"
> > true "DPDK 17.11.4" {hostname="v41",
> > rundir="/var/run/openvswitch",
> > system-id="7d5d1db0-87db-41bc-bb94-661dde831e9f"} [dpdk, dpdkr,
> > dpdkvhostuser, dpdkvhostum
> >
> >
> > Port table
> >
> > _uuid bond_active_slave bond_downdelay
> > bond_fake_iface bond_mode bond_updelay cvlans external_ids
> fake_bridge
> > interfaces lacp mac name other_config
> > protected qos rstp_statistics rstp_status statistics status tag
> trunks
> > vlan_mode
> >
> > ------------------------------------ ----------------- --------------
> > --------------- --------- ------------ ------ ------------
> -----------
> > -------------------------------------- ---- --- ------ ------------
> > --------- --- --------------- ----------- ---------- ------ ---
> ------
> > ---------
> >
> > 60b0661f-2177-4283-a6cb-a80336a8c478 [] 0
> > false [] 0 [] {} false
> > [271077eb-97af-4c2d-a5ee-2c63c9367312] [] [] "br0" {}
> > false [] {} {} {} {} []
> [] []
> >
> > 3bd79dba-777c-40d0-b573-bf9e027326f4 [] 0
> > false [] 0 [] {} false
> > [4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136] [] [] "p2p1" {}
> > false [] {} {} {} {} []
> [] []
> >
> >
> > QoS table
> >
> > _uuid external_ids other_config queues type
> >
> > ----- ------------ ------------ ------ ----
> >
> >
> > Queue table
> >
> > _uuid dscp external_ids other_config
> >
> > ----- ---- ------------ ------------
> >
> >
> > SSL table
> >
> > _uuid bootstrap_ca_cert ca_cert certificate external_ids private_key
> >
> > ----- ----------------- ------- ----------- ------------ -----------
> >
> >
> > sFlow table
> >
> > _uuid agent external_ids header polling sampling targets
> >
> > ----- ----- ------------ ------ ------- -------- -------
> >
> >
> > I see rx_dropped counters. Other than that I'm not sure what's
> going on,
> > as I've tried various tuning suggested by dpdk, Intel and Red
> Hat, but
> > performance remains unimproved.
> >
> >
> > Regards,
> >
> >
> > Rob
> >
> >
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org <mailto:discuss at openvswitch.org>
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
> >
>
>
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
More information about the discuss
mailing list