<div dir="ltr">Hi Tiago,<div><br></div><div>you are correct, this is a virtualization environment. The tests numbers I am quoting are running directly on the hypervisor, rather than guests.</div><div><br></div>My ultimate aim is to get good performance for guests talking over VXLAN. Using OvS I get around 5Gb/s between guests talking over VXLAN and near wire speed performance where the guests talk to a bridge without VXLAN. I'd like to improve on the VXLAN performance.<div><br><div><div>I believe VXLAN is offloaded by tx-udp_tnl-segmentation? (...VXLAN is being configured at the interface level by OpenNebula, so handled by the kernel module).</div><div><br><div>Regarding the NIC, I see:</div><div><br></div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">[root@v41 ~]# ethtool -k p2p1 | grep tcp-segmentation-offload</span></p>
<p class="gmail-p2" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(195,55,32)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><b>tcp-segmentation-offload</b></span><span class="gmail-s2" style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">: on</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><br></p>Are the TSO patches for OvS-DPDK suitable to apply and use, if so anything I should know? eg version to apply to, source to obtain current patches from?</div><div><br></div><div>Many thanks,</div><div><br></div><div>Rob</div><div><br></div><div><div class="gmail_quote"><div dir="ltr">On Fri, Nov 16, 2018 at 2:06 AM Lam, Tiago <<a href="mailto:tiago.lam@intel.com">tiago.lam@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I have a couple of questions, but at a first glance this seems to be a<br>
result of OvS-DPDK not yet supporting TSO, while the Kernel does. We are<br>
in the works of addressing that.<br>
<br>
The numbers you are reporting, ~9.4 Gbps vs ~5.5 Gbps are very similar<br>
to the ones we have been getting with and without our TSO series [1]<br>
applied.<br>
<br>
On the description below, is this a virtualized environment by any<br>
change? I.e. are you running the iperf tests on top of VMs running on<br>
those Centos hosts, or directly on them?<br>
<br>
Can you also confirm you are using TSO when using OvS with the Kernel by<br>
reporting what you get with $ethtool -k $iface | grep<br>
"tcp-segmentation-offload:". Is it reported as "on"?<br>
<br>
Regards,<br>
Tiago.<br>
<br>
[1] <a href="https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html" rel="noreferrer" target="_blank">https://mail.openvswitch.org/pipermail/ovs-dev/2018-August/350832.html</a><br>
<br>
On 15/11/2018 22:17, Robert Brooks wrote:<br>
> I have a pair of Centos 7 hosts on 10gig switch.<br>
> <br>
> Using OVS without dpdk enabled I can get 9.4Gb/s with a simple iperf test.<br>
> <br>
> After switching the receiving host to ovs-dpdk following guidance here:<br>
> <a href="http://docs.openvswitch.org/en/latest/intro/install/dpdk/" rel="noreferrer" target="_blank">http://docs.openvswitch.org/en/latest/intro/install/dpdk/</a><br>
> <br>
> I get 1.02Gb/s with 1500 MTU and 5.75Gb/s with a 9000 MTU.<br>
> <br>
> Hardware is a Dell R630, with two E5-2680 26 core @ 2.40GHz cpus, 256GB<br>
> RAM, Intel 92599ES NIC.<br>
> <br>
> I have confirmed the doucmented kernel boot options are set and 1GB<br>
> hugepages are in use.<br>
> <br>
> # cat /proc/cmdline <br>
> BOOT_IMAGE=/vmlinuz-3.10.0-862.3.2.el7.x86_64<br>
> root=UUID=7d4edcad-0fd5-4224-a55f-ff2db81aac27 ro crashkernel=auto<br>
> <a href="http://rd.lvm.lv" rel="noreferrer" target="_blank">rd.lvm.lv</a> <<a href="http://rd.lvm.lv" rel="noreferrer" target="_blank">http://rd.lvm.lv</a>>=vg01/swap <a href="http://rd.lvm.lv" rel="noreferrer" target="_blank">rd.lvm.lv</a><br>
> <<a href="http://rd.lvm.lv" rel="noreferrer" target="_blank">http://rd.lvm.lv</a>>=vg01/usr console=ttyS0,115200 iommu=pt intel_iommu=on<br>
> default_hugepagesz=1G hugepagesz=1G hugepages=8<br>
> <br>
> Software is openvswitch 2.10.1 built using the provided rpm spec file. I<br>
> have used both the Centos provided dpdk 17.11 and a re-compile using<br>
> latest LTS of 17.11.4.<br>
> <br>
> I have tried various performance tweaks, including cpu pinning and<br>
> isolated cpus.<br>
> <br>
> Switching back to regular ovs returns the performance to close to wire<br>
> speed.<br>
> <br>
> Under load the following is logged:<br>
> <br>
> 2018-11-15T21:59:24.306Z|00170|poll_loop|INFO|wakeup due to [POLLIN] on<br>
> fd 74 (character device /dev/net/tun) at lib/netdev-linux.c:1347 (98%<br>
> CPU usage)<br>
> <br>
> Config dump:<br>
> <br>
> # ovsdb-client dump Open_vSwitch<br>
> <br>
> AutoAttach table<br>
> <br>
> _uuid mappings system_description system_name<br>
> <br>
> ----- -------- ------------------ -----------<br>
> <br>
> <br>
> Bridge table<br>
> <br>
> _uuid auto_attach controller datapath_id <br>
> datapath_type datapath_version external_ids fail_mode flood_vlans<br>
> flow_tables ipfix mcast_snooping_enable mirrors name netflow<br>
> other_config ports <br>
> <br>
> <br>
> ------------------------------------ ----------- ----------<br>
> ------------------ ------------- ---------------- ------------ ---------<br>
> ----------- ----------- ----- --------------------- ------- -----<br>
> ------- ------------<br>
> ---------------------------------------------------------------------<br>
> <br>
> 006f021a-b14d-49fb-a11d-48df2fa2bca1 [] [] <br>
> "0000001b21a6ddc4" netdev "<built-in>" {} [] <br>
> [] {} [] false [] "br0" [] <br>
> {} [3bd79dba-777c-40d0-b573-bf9e027326f4,<br>
> 60b0661f-2177-4283-a6cb-a80336<br>
> <br>
> <br>
> Controller table<br>
> <br>
> _uuid connection_mode controller_burst_limit controller_rate_limit<br>
> enable_async_messages external_ids inactivity_probe is_connected<br>
> local_gateway local_ip local_netmask max_backoff other_config role<br>
> status target<br>
> <br>
> ----- --------------- ---------------------- ---------------------<br>
> --------------------- ------------ ---------------- ------------<br>
> ------------- -------- ------------- ----------- ------------ ----<br>
> ------ ------<br>
> <br>
> <br>
> Flow_Sample_Collector_Set table<br>
> <br>
> _uuid bridge external_ids id ipfix<br>
> <br>
> ----- ------ ------------ -- -----<br>
> <br>
> <br>
> Flow_Table table<br>
> <br>
> _uuid external_ids flow_limit groups name overflow_policy prefixes<br>
> <br>
> ----- ------------ ---------- ------ ---- --------------- --------<br>
> <br>
> <br>
> IPFIX table<br>
> <br>
> _uuid cache_active_timeout cache_max_flows external_ids obs_domain_id<br>
> obs_point_id other_config sampling targets<br>
> <br>
> ----- -------------------- --------------- ------------ -------------<br>
> ------------ ------------ -------- -------<br>
> <br>
> <br>
> Interface table<br>
> <br>
> _uuid admin_state bfd bfd_status<br>
> cfm_fault cfm_fault_status cfm_flap_count cfm_health cfm_mpid<br>
> cfm_remote_mpids cfm_remote_opstate duplex error external_ids ifindex <br>
> ingress_policing_burst ingress_policing_rate lacp_current link_resets<br>
> link_speed link_stat <br>
> <br>
> ------------------------------------ ----------- --- ----------<br>
> --------- ---------------- -------------- ---------- --------<br>
> ---------------- ------------------ ------ ----- ------------ --------<br>
> ---------------------- --------------------- ------------ -----------<br>
> ----------- ----------<br>
> <br>
> 271077eb-97af-4c2d-a5ee-2c63c9367312 up {} {} [] <br>
> [] [] [] [] [] <br>
> [] full [] {} 13 0 <br>
> 0 [] 11 10000000 up <br>
> <br>
> 4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136 up {} {} [] <br>
> [] [] [] [] [] <br>
> [] full [] {} 13032858 0 <br>
> 0 [] 0 10000000000 up s<br>
> <br>
> <br>
> Manager table<br>
> <br>
> _uuid connection_mode external_ids inactivity_probe is_connected<br>
> max_backoff other_config status target<br>
> <br>
> ----- --------------- ------------ ---------------- ------------<br>
> ----------- ------------ ------ ------<br>
> <br>
> <br>
> Mirror table<br>
> <br>
> _uuid external_ids name output_port output_vlan select_all<br>
> select_dst_port select_src_port select_vlan snaplen statistics<br>
> <br>
> ----- ------------ ---- ----------- ----------- ----------<br>
> --------------- --------------- ----------- ------- ----------<br>
> <br>
> <br>
> NetFlow table<br>
> <br>
> _uuid active_timeout add_id_to_interface engine_id engine_type<br>
> external_ids targets<br>
> <br>
> ----- -------------- ------------------- --------- -----------<br>
> ------------ -------<br>
> <br>
> <br>
> Open_vSwitch table<br>
> <br>
> _uuid bridges <br>
> cur_cfg datapath_types db_version dpdk_initialized dpdk_version <br>
> external_ids <br>
> iface_types <br>
> <br>
> ------------------------------------<br>
> -------------------------------------- ------- ----------------<br>
> ---------- ---------------- --------------<br>
> --------------------------------------------------------------------------------------------------------<br>
> ----------------------------------------<br>
> <br>
> ea8e7bcc-df11-42bd-96c9-b7eaafdfa863<br>
> [006f021a-b14d-49fb-a11d-48df2fa2bca1] 45 [netdev, system] "7.16.1"<br>
> true "DPDK 17.11.4" {hostname="v41",<br>
> rundir="/var/run/openvswitch",<br>
> system-id="7d5d1db0-87db-41bc-bb94-661dde831e9f"} [dpdk, dpdkr,<br>
> dpdkvhostuser, dpdkvhostum<br>
> <br>
> <br>
> Port table<br>
> <br>
> _uuid bond_active_slave bond_downdelay<br>
> bond_fake_iface bond_mode bond_updelay cvlans external_ids fake_bridge<br>
> interfaces lacp mac name other_config<br>
> protected qos rstp_statistics rstp_status statistics status tag trunks<br>
> vlan_mode<br>
> <br>
> ------------------------------------ ----------------- --------------<br>
> --------------- --------- ------------ ------ ------------ -----------<br>
> -------------------------------------- ---- --- ------ ------------<br>
> --------- --- --------------- ----------- ---------- ------ --- ------<br>
> ---------<br>
> <br>
> 60b0661f-2177-4283-a6cb-a80336a8c478 [] 0 <br>
> false [] 0 [] {} false <br>
> [271077eb-97af-4c2d-a5ee-2c63c9367312] [] [] "br0" {} <br>
> false [] {} {} {} {} [] [] []<br>
> <br>
> 3bd79dba-777c-40d0-b573-bf9e027326f4 [] 0 <br>
> false [] 0 [] {} false <br>
> [4c3f7cdb-fd16-44c7-bb82-4aa8cef0c136] [] [] "p2p1" {} <br>
> false [] {} {} {} {} [] [] []<br>
> <br>
> <br>
> QoS table<br>
> <br>
> _uuid external_ids other_config queues type<br>
> <br>
> ----- ------------ ------------ ------ ----<br>
> <br>
> <br>
> Queue table<br>
> <br>
> _uuid dscp external_ids other_config<br>
> <br>
> ----- ---- ------------ ------------<br>
> <br>
> <br>
> SSL table<br>
> <br>
> _uuid bootstrap_ca_cert ca_cert certificate external_ids private_key<br>
> <br>
> ----- ----------------- ------- ----------- ------------ -----------<br>
> <br>
> <br>
> sFlow table<br>
> <br>
> _uuid agent external_ids header polling sampling targets<br>
> <br>
> ----- ----- ------------ ------ ------- -------- -------<br>
> <br>
> <br>
> I see rx_dropped counters. Other than that I'm not sure what's going on,<br>
> as I've tried various tuning suggested by dpdk, Intel and Red Hat, but<br>
> performance remains unimproved.<br>
> <br>
> <br>
> Regards,<br>
> <br>
> <br>
> Rob<br>
> <br>
> <br>
> _______________________________________________<br>
> discuss mailing list<br>
> <a href="mailto:discuss@openvswitch.org" target="_blank">discuss@openvswitch.org</a><br>
> <a href="https://mail.openvswitch.org/mailman/listinfo/ovs-discuss" rel="noreferrer" target="_blank">https://mail.openvswitch.org/mailman/listinfo/ovs-discuss</a><br>
> <br>
</blockquote></div></div></div></div></div></div>