[ovs-discuss] ovs crash when running traffic from VM to VM over DPDK and vhostuser

Yi Ba yby.developer at yahoo.com
Mon Apr 4 21:17:39 UTC 2016


   
   - What you did that make the problem appear.   
We have an openstack kilo setup. it has 3 controllers and 3 computes. 1 of the controllers runs an ODL, which manages the OVS on each compute host. The compute hosts are running an hlinux OS, which is HPE's Debian8-based OS.   
each host has 2 numa nodes, each with 12 cores (24 Hyper Threaded). each numa with 64GB.   
We patched neutron to create vhostuser ports (which is not available in stable kilo), in order to work with dpdk in order to achieve highest throughput possible.   
OVS was running with "-c 4" and pmd-core-mask 0x38. all these cores were isolated.   
nova was configured with vcpu_pin_set=6-11, and the flavor had 6 vCPUs. flavor had 16 1GB huge pages, backed up by real 1GB huge pages in host.   
Then running a traffic generator inside 2 VMs, using DPDK, in order to generate traffic. sending directly to the other VMs mac and IP.   

   - What you expected to happen.   
We expected traffic to flow.   

   - What actually happened.   
OVS crashed (in dpdk code). Attached BT.   
   

   
   - The Open vSwitch version number (as output by ovs-vswitchd --version)   
root at BASE-CCP-CPN-N0001-NETCLM:~# ovs-vswitchd --version   
ovs-vswitchd (Open vSwitch) 2.5.0   
Compiled Apr  4 2016 08:51:09   

   - Any local patches or changes you have applied (if any).   
applied ce179f1163f947fe8dc5afa35a2cdd0756bb53a0   

The following are also handy sometimes:   
   - The kernel version on which Open vSwitch is running (from /proc/version) and the distribution and version number of your OS (e.g. "Centos 5.0").   
root at BASE-CCP-CPN-N0001-NETCLM:~# cat /proc/version   
Linux version 3.14.48-1-amd64-hlinux (pbuilder at build) (gcc version 4.9.2 (Debian 4.9.2-10) ) #hlinux1 SMP Thu Aug 6 16:02:22 UTC 2015   

   - If you have Open vSwitch configured to connect to an OpenFlow controller, the output of ovs-ofctl show <bridge> for each <bridge> configured in the vswitchd configuration database.   
We are using odl. attached outputs.   

   - A fix or workaround, if you have one   
We disabled mrg_rxbuf (mrg_rxbuf=off) in qemu   


We can supply more info if necessary, like our exact build process etc.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160404/c86c8765/attachment-0002.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ovs-ofctl.txt
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160404/c86c8765/attachment-0004.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: ovs-vswitchd-gdb.txt
URL: <http://openvswitch.org/pipermail/ovs-discuss/attachments/20160404/c86c8765/attachment-0005.txt>


More information about the discuss mailing list