[ovs-dev] [PATCH RFC] - vxlan udp csum and openstack ovs performance

Suryanarayan Ramamurthy ramu.ramamurthy at us.ibm.com
Fri Aug 28 00:39:57 UTC 2015


Overview:

    We are running openstack kilo with neutron OVS using intel 10G adapters (br kx4 dual-port, 82599es).
neutron OVS is using vxlan tunnels. We are using RHEL 7.1 (which uses the 3.x kernel). 
Openvswitch kernel module is from upstream (3.x) and ovs is 2.3.0 based. 

Problem:

We find that the aggregate throughput does not exceed 3Gbps on the 10Gb link
when running various VM test scenarios (40VMs<-->40VMs). 
The root cause is the inability to perform GRO at the physical device for the vxlan protocol.

See discussion on this problem and its follow ups:
http://marc.info/?t=143527719300002&r=1&w=2

Fix:

We found that enabling UDP checksums for vxlan triggers GRO on the receiver, and boosts the vxlan performance to 8+ Gbps.
Hence, we want to enable vxlan csum for OVS vxlan tunnels. This is already supported in OVS 2.4, and kernel versions 4+ - But we 
want this fix to appear in 3.x upstream kernels, and the corresponding userspace change in OVS 2.3.

So, the following set of patches achieves 8+Gbps throughput on 10G adapters (vs the 2Gbps without these) on
the above Openstack topology.

1)
   Backport this patch (to ovs 2.3) which allows the csum configuration on vxlan ports
 
http://openvswitch.org/pipermail/dev/2015-March/052823.html

2) Apply this new patch to send the CSUM attribute to the kernel module

--- /home/ovs/tmp/openvswitch-2.3.0/lib/dpif-linux.c	2014-08-14 16:34:33.000000000 -0400
+++ lib/dpif-linux.c	2015-08-27 18:56:27.282610654 -0400
@@ -666,10 +666,15 @@
     }
 
     tnl_cfg = netdev_get_tunnel_config(netdev);
-    if (tnl_cfg && tnl_cfg->dst_port != 0) {
+    if (tnl_cfg && (tnl_cfg->dst_port != 0 || tnl_cfg->csum)) {
         ofpbuf_use_stack(&options, options_stub, sizeof options_stub);
-        nl_msg_put_u16(&options, OVS_TUNNEL_ATTR_DST_PORT,
-                       ntohs(tnl_cfg->dst_port));
+        if (tnl_cfg->dst_port != 0) {
+            nl_msg_put_u16(&options, OVS_TUNNEL_ATTR_DST_PORT,
+                           ntohs(tnl_cfg->dst_port));
+        }
+        if (tnl_cfg->csum) {
+            nl_msg_put_flag(&options, OVS_TUNNEL_KEY_ATTR_CSUM);
+        }
         request.options = ofpbuf_data(&options);
         request.options_len = ofpbuf_size(&options);
     }

3) Apply this patch to the upstream 3.x kernels to set the UDP csum flag when creating the vxlan socket

--- ../linux-3.10.0-229.el7/net/openvswitch/vport-vxlan.c	2015-01-29 18:15:53.000000000 -0500
+++ vport-vxlan.c	2015-08-27 18:51:35.619688332 -0400
@@ -100,6 +100,7 @@
 	struct nlattr *a;
 	u16 dst_port;
 	int err;
+	u32 vxlan_sock_flags = 0;
 
 	if (!options) {
 		err = -EINVAL;
@@ -114,6 +115,11 @@
 		goto error;
 	}
 
+	a = nla_find_nested(options, OVS_TUNNEL_KEY_ATTR_CSUM);
+	if (a) {
+		vxlan_sock_flags |= VXLAN_F_UDP_CSUM;
+	} 
+
 	vport = ovs_vport_alloc(sizeof(struct vxlan_port),
 				&ovs_vxlan_vport_ops, parms);
 	if (IS_ERR(vport))
@@ -122,7 +128,7 @@
 	vxlan_port = vxlan_vport(vport);
 	strncpy(vxlan_port->name, parms->name, IFNAMSIZ);
 
-	vs = vxlan_sock_add(net, htons(dst_port), vxlan_rcv, vport, true, 0);
+	vs = vxlan_sock_add(net, htons(dst_port), vxlan_rcv, vport, true, vxlan_sock_flags);
 	if (IS_ERR(vs)) {
 		ovs_vport_free(vport);
 		return (void *)vs;


4) Patch Openstack neutron-ovs-agent to optionally set the csum flag on vxlan tunnels 
(this will be taken to the Openstack Neutron community)

Conclusion:

Since above patches provide a huge performance boost to OVS-based Openstack neutron using vxlan tunnels on 
widely deployed 10G adapters, we want the community feedback on how best to apply the above patches.








More information about the dev mailing list