[ovs-discuss] ovs-vswitchd SEGV when using vhost-user feature with dpdk
Loftus, Ciara
ciara.loftus at intel.com
Fri Oct 9 07:07:17 UTC 2015
>
> I have ran into a ovs-vswitchd SEGV while trying to use the vhost-user
> feature with the latest dpdk. I have attempted to provide as much detail as I
> can.
>
> versions compiled from source on CentOS 7.1 x86_64:
> - openvswitch 2.4.0 - http://openvswitch.org/releases/openvswitch-
> 2.4.0.tar.gz
> - dpdk 2.1.0 - http://dpdk.org/browse/dpdk/snapshot/dpdk-2.1.0.tar.gz
> - qemu - 2.4.0.1 - http://wiki.qemu-project.org/download/qemu-
> 2.4.0.1.tar.bz2
>
>
> Steps to reproduce:
>
> - Build and Install following these instructions
> http://wiki.qemu.org/Features/vhost-user-ovs-dpdk
> - setup details:
>
> hugepage config:
> GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap
> crashkernel=auto rhgb quiet iommu=pt intel_iommu=on
> default_hugepagesz=1G hugepagesz=1G hugepages=8
> isolcpus=1,2,3,4,5,12,13,14,15,16,17,7,8,9,10,11,18,19,20,21,22,23
> transparent_hugepage=never"
>
> # export DPDK_DIR=/root/dpdk/dpdk-2.1.0
> # export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/
> # export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
>
>
> # modprobe openvswitch
> # modprobe uio
> # insmod /root/dpdk/dpdk-2.1.0/x86_64-native-linuxapp-
> gcc/kmod/igb_uio.ko
> # insmod /root/dpdk/dpdk-
> 2.1.0/lib/librte_vhost/eventfd_link/eventfd_link.ko
>
> # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f0
> # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f1
>
> # /root/openvswitch/openvswitch-2.4.0/ovsdb/ovsdb-server --
> remote=punix:/usr/local/var/run/openvswitch/db.sock --
> remote=db:Open_vSwitch,Open_vSwitch,manager_options --pidfile --
> detach
>
> # rm -f /usr/local/var/run/openvswitch/vhost-user*
>
> # /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-vswitchd --dpdk -c
> 0xffff -n 4 --socket-mem 1024 -- unix:$DB_SOCK --pidfile --detach --log-
> file=/var/log/openvswitch/ovs-vswitchd.log
>
> # lspci | grep Ethernet
> 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit
> X540-AT2 (rev 01)
> 01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit
> X540-AT2 (rev 01)
>
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-br br0 -- set
> bridge br0 datapath_type=netdev
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> dpdk0 -- set Interface dpdk0 type=dpdk
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> dpdk1 -- set Interface dpdk1 type=dpdk
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser
> # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
>
> - Start the first VM like this:
> ./qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 \
My first suggestion would be to try supplying your VM with more memory. ie. instead of "-m 1024", try "-m 4096"
Thanks,
Ciara
> -display sdl \
> -sdl \
> -chardev socket,id=char0,path=/usr/local/var/run/openvswitch/vhost-user1
> \
> -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce \
> -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01 \
> -object memory-backend-file,id=mem,size=1024M,mem-
> path=/dev/hugepages,share=on \
> -numa node,memdev=mem -mem-prealloc \
> -net user,hostfwd=tcp::10021-:22 \
> -net nic \
> /var/lib/libvirt/images/centos-vm1.qcow2 &
>
> - From another machine send iperf traffic to this VM.
> iperf -s (in the VM)
> iperf -c <ipaddr> -t 60 -P 12 (from another machine)
>
> - After about a few seconds, iperf stalls. network connectivity breaks. Can't
> ping the VM anymore.
> - Saw these in the ovs-vswitchd log files.
>
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> Failed to allocate memory for mbuf.
>
> - shutdown iperf ... then shutdown the VM.
> - at this point ovs-vswitchd crashes with the following SEGV.
>
> [root at localhost vswitchd]# gdb ./ovs-vswitchd /var/spool/abrt/ccpp-2015-
> 09-29-18\:47\:37-22012/coredump
> GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
> Copyright (C) 2013 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-
> vswitchd...done.
> [New LWP 22051]
> [New LWP 22012]
> [New LWP 22013]
> [New LWP 22015]
> [New LWP 22039]
> [New LWP 22040]
> [New LWP 22041]
> [New LWP 22042]
> [New LWP 22043]
> [New LWP 22044]
> [New LWP 22045]
> [New LWP 22046]
> [New LWP 22047]
> [New LWP 22048]
> [New LWP 22049]
> [New LWP 22050]
> [New LWP 22014]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/root/openvswitch/openvswitch-
> 2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4'.
> Program terminated with signal 11, Segmentation fault.
> #0 0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> /lib64/libdpdk.so
> Missing separate debuginfos, use: debuginfo-install glibc-2.17-78.el7.x86_64
> (gdb) where
> #0 0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> /lib64/libdpdk.so
> #1 0x000000000056d103 in netdev_dpdk_vhost_rxq_recv
> (rxq_=0x7efe5a537600, packets=0x7eff4c9a2930, c=0x7eff4c9a292c)
> at lib/netdev-dpdk.c:952
> #2 0x00000000004a2dec in netdev_rxq_recv (rx=0x7efe5a537600,
> buffers=0x7eff4c9a2930, cnt=0x7eff4c9a292c)
> at lib/netdev.c:652
> #3 0x0000000000478f33 in dp_netdev_process_rxq_port (pmd=0x231c310,
> port=0x2166990, rxq=0x7efe5a537600)
> at lib/dpif-netdev.c:2520
> #4 0x0000000000479648 in pmd_thread_main (f_=0x231c310) at lib/dpif-
> netdev.c:2680
> #5 0x00000000004f60f3 in ovsthread_wrapper (aux_=0x231c150) at lib/ovs-
> thread.c:340
> #6 0x00007eff4d276df5 in start_thread () from /lib64/libpthread.so.0
> #7 0x00007eff4ca9a1ad in clone () from /lib64/libc.so.6
More information about the discuss
mailing list