[ovs-dev] [PATCH RFC] - vxlan udp csum and openstack ovs performance

[Ramu Ramamurthy]

posting again, since the earlier emailer mangles email addresses.

Overview:

     We are running openstack kilo with neutron OVS using intel 10G 
adapters (br kx4 dual-port, 82599es).
neutron OVS is using vxlan tunnels. We are using RHEL 7.1 (which uses 
the 3.x kernel).
Openvswitch kernel module is from upstream (3.x) and ovs is 2.3.0 based.

Problem:

We find that the aggregate throughput does not exceed 3Gbps on the 10Gb 
link
when running various VM test scenarios (40VMs<-->40VMs).
The root cause is the inability to perform GRO at the physical device 
for the vxlan protocol.

See discussion on this problem and its follow ups:
http://marc.info/?t=143527719300002&r=1&w=2

Fix:

We found that enabling UDP checksums for vxlan triggers GRO on the 
receiver, and boosts the vxlan performance to 8+ Gbps.
Hence, we want to enable vxlan csum for OVS vxlan tunnels. This is 
already supported in OVS 2.4, and kernel versions 4+ - But we
want this fix to appear in 3.x upstream kernels, and the corresponding 
userspace change in OVS 2.3.

So, the following set of patches achieves 8+Gbps throughput on 10G 
adapters (vs the 2Gbps without these) on
the above Openstack topology.

1)
    Backport this patch (to ovs 2.3) which allows the csum configuration 
on vxlan ports

http://openvswitch.org/pipermail/dev/2015-March/052823.html

2) Apply this new patch to send the CSUM attribute to the kernel module

--- /home/ovs/tmp/openvswitch-2.3.0/lib/dpif-linux.c 2014-08-14 
16:34:33.000000000 -0400
+++ lib/dpif-linux.c 2015-08-27 18:56:27.282610654 -0400
@@ -666,10 +666,15 @@
      }

      tnl_cfg = netdev_get_tunnel_config(netdev);
-    if (tnl_cfg && tnl_cfg->dst_port != 0) {
+    if (tnl_cfg && (tnl_cfg->dst_port != 0 || tnl_cfg->csum)) {
          ofpbuf_use_stack(&options, options_stub, sizeof options_stub);
-        nl_msg_put_u16(&options, OVS_TUNNEL_ATTR_DST_PORT,
-                       ntohs(tnl_cfg->dst_port));
+        if (tnl_cfg->dst_port != 0) {
+            nl_msg_put_u16(&options, OVS_TUNNEL_ATTR_DST_PORT,
+                           ntohs(tnl_cfg->dst_port));
+        }
+        if (tnl_cfg->csum) {
+            nl_msg_put_flag(&options, OVS_TUNNEL_KEY_ATTR_CSUM);
+        }
          request.options = ofpbuf_data(&options);
          request.options_len = ofpbuf_size(&options);
      }

3) Apply this patch to the upstream 3.x kernels to set the UDP csum flag 
when creating the vxlan socket

--- ../linux-3.10.0-229.el7/net/openvswitch/vport-vxlan.c 2015-01-29 
18:15:53.000000000 -0500
+++ vport-vxlan.c 2015-08-27 18:51:35.619688332 -0400
@@ -100,6 +100,7 @@
   struct nlattr *a;
   u16 dst_port;
   int err;
+ u32 vxlan_sock_flags = 0;

   if (!options) {
   err = -EINVAL;
@@ -114,6 +115,11 @@
   goto error;
   }

+ a = nla_find_nested(options, OVS_TUNNEL_KEY_ATTR_CSUM);
+ if (a) {
+ vxlan_sock_flags |= VXLAN_F_UDP_CSUM;
+ }
+
   vport = ovs_vport_alloc(sizeof(struct vxlan_port),
   &ovs_vxlan_vport_ops, parms);
   if (IS_ERR(vport))
@@ -122,7 +128,7 @@
   vxlan_port = vxlan_vport(vport);
   strncpy(vxlan_port->name, parms->name, IFNAMSIZ);

- vs = vxlan_sock_add(net, htons(dst_port), vxlan_rcv, vport, true, 0);
+ vs = vxlan_sock_add(net, htons(dst_port), vxlan_rcv, vport, true, 
vxlan_sock_flags);
   if (IS_ERR(vs)) {
   ovs_vport_free(vport);
   return (void *)vs;


4) Patch Openstack neutron-ovs-agent to optionally set the csum flag on 
vxlan tunnels
(this will be taken to the Openstack Neutron community)

Conclusion:

Since above patches provide a huge performance boost to OVS-based 
Openstack neutron using vxlan tunnels on
widely deployed 10G adapters, we want the community feedback on how best 
to apply the above patches.