[ovs-discuss] ovs-vswitchd SEGV when using vhost-user feature with dpdk

Loftus, Ciara ciara.loftus at intel.com
Fri Oct 9 07:07:17 UTC 2015


> 
> I have ran into a ovs-vswitchd SEGV while trying to use the vhost-user
> feature with the latest dpdk. I have attempted to provide as much detail as I
> can.
> 
> versions compiled from source on CentOS 7.1 x86_64:
>  - openvswitch 2.4.0 - http://openvswitch.org/releases/openvswitch-
> 2.4.0.tar.gz
>  - dpdk 2.1.0 - http://dpdk.org/browse/dpdk/snapshot/dpdk-2.1.0.tar.gz
>  - qemu - 2.4.0.1 - http://wiki.qemu-project.org/download/qemu-
> 2.4.0.1.tar.bz2
> 
> 
> Steps to reproduce:
> 
> - Build and Install following these instructions
> http://wiki.qemu.org/Features/vhost-user-ovs-dpdk
> - setup details:
> 
> hugepage config:
> GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap
> crashkernel=auto rhgb quiet iommu=pt intel_iommu=on
> default_hugepagesz=1G hugepagesz=1G hugepages=8
> isolcpus=1,2,3,4,5,12,13,14,15,16,17,7,8,9,10,11,18,19,20,21,22,23
> transparent_hugepage=never"
> 
> # export DPDK_DIR=/root/dpdk/dpdk-2.1.0
> # export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/
> # export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> 
> 
> # modprobe openvswitch
> # modprobe uio
> # insmod /root/dpdk/dpdk-2.1.0/x86_64-native-linuxapp-
> gcc/kmod/igb_uio.ko
> # insmod /root/dpdk/dpdk-
> 2.1.0/lib/librte_vhost/eventfd_link/eventfd_link.ko
> 
> # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f0
> # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f1
> 
> # /root/openvswitch/openvswitch-2.4.0/ovsdb/ovsdb-server --
> remote=punix:/usr/local/var/run/openvswitch/db.sock     --
> remote=db:Open_vSwitch,Open_vSwitch,manager_options     --pidfile --
> detach
> 
> # rm -f /usr/local/var/run/openvswitch/vhost-user*
> 
> # /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-vswitchd --dpdk -c
> 0xffff -n 4 --socket-mem 1024 -- unix:$DB_SOCK --pidfile --detach --log-
> file=/var/log/openvswitch/ovs-vswitchd.log
> 
> # lspci | grep Ethernet
> 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit
> X540-AT2 (rev 01)
> 01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit
> X540-AT2 (rev 01)
> 
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-br br0 -- set
> bridge br0 datapath_type=netdev
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> dpdk0 -- set Interface dpdk0 type=dpdk
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> dpdk1 -- set Interface dpdk1 type=dpdk
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
> 
> - Start the first VM like this:
> ./qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 \

My first suggestion would be to try supplying your VM with more memory. ie. instead of "-m 1024", try "-m 4096"

Thanks,
Ciara

>  -display sdl \
>  -sdl \
>  -chardev socket,id=char0,path=/usr/local/var/run/openvswitch/vhost-user1
> \
>  -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce  \
>  -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01 \
>  -object memory-backend-file,id=mem,size=1024M,mem-
> path=/dev/hugepages,share=on \
>  -numa node,memdev=mem -mem-prealloc \
>  -net user,hostfwd=tcp::10021-:22 \
>  -net nic  \
>  /var/lib/libvirt/images/centos-vm1.qcow2  &
> 
> - From another machine send iperf traffic to this VM.
>   iperf -s (in the VM)
>   iperf -c <ipaddr> -t 60 -P 12 (from another machine)
> 
> - After about a few seconds, iperf stalls. network connectivity breaks. Can't
> ping the VM anymore.
> - Saw these in the ovs-vswitchd log files.
> 
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> 
> - shutdown iperf ... then shutdown the VM.
> - at this point ovs-vswitchd crashes with the following SEGV.
> 
> [root at localhost vswitchd]# gdb ./ovs-vswitchd /var/spool/abrt/ccpp-2015-
> 09-29-18\:47\:37-22012/coredump
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-
> vswitchd...done.
> [New LWP 22051]
> [New LWP 22012]
> [New LWP 22013]
> [New LWP 22015]
> [New LWP 22039]
> [New LWP 22040]
> [New LWP 22041]
> [New LWP 22042]
> [New LWP 22043]
> [New LWP 22044]
> [New LWP 22045]
> [New LWP 22046]
> [New LWP 22047]
> [New LWP 22048]
> [New LWP 22049]
> [New LWP 22050]
> [New LWP 22014]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/root/openvswitch/openvswitch-
> 2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> /lib64/libdpdk.so
> Missing separate debuginfos, use: debuginfo-install glibc-2.17-78.el7.x86_64
> (gdb) where
> #0  0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> /lib64/libdpdk.so
> #1  0x000000000056d103 in netdev_dpdk_vhost_rxq_recv
> (rxq_=0x7efe5a537600, packets=0x7eff4c9a2930, c=0x7eff4c9a292c)
>     at lib/netdev-dpdk.c:952
> #2  0x00000000004a2dec in netdev_rxq_recv (rx=0x7efe5a537600,
> buffers=0x7eff4c9a2930, cnt=0x7eff4c9a292c)
>     at lib/netdev.c:652
> #3  0x0000000000478f33 in dp_netdev_process_rxq_port (pmd=0x231c310,
> port=0x2166990, rxq=0x7efe5a537600)
>     at lib/dpif-netdev.c:2520
> #4  0x0000000000479648 in pmd_thread_main (f_=0x231c310) at lib/dpif-
> netdev.c:2680
> #5  0x00000000004f60f3 in ovsthread_wrapper (aux_=0x231c150) at lib/ovs-
> thread.c:340
> #6  0x00007eff4d276df5 in start_thread () from /lib64/libpthread.so.0
> #7  0x00007eff4ca9a1ad in clone () from /lib64/libc.so.6


More information about the discuss mailing list