[ovs-discuss] ovs-vswitchd SEGV when using vhost-user feature with dpdk

Loftus, Ciara ciara.loftus at intel.com
Mon Oct 12 10:49:31 UTC 2015


> 
> Increasing qemu to -m 4096, causes an immediate ovs-vswitchd SEGV
> 
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib64/libthread_db.so.1".
> Core was generated by `/root/openvswitch/openvswitch-
> 2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007fca43ed6f2c in set_vring_addr () from /lib64/libdpdk.so
> (gdb) where
> #0  0x00007fca43ed6f2c in set_vring_addr () from /lib64/libdpdk.so
> #1  0x00007fca43f90d2e in vserver_message_handler () from
> /lib64/libdpdk.so
> #2  0x00007fca43f3ad7d in fdset_event_dispatch () from /lib64/libdpdk.so
> #3  0x00007fca43f91330 in rte_vhost_driver_session_start () from
> /lib64/libdpdk.so
> #4  0x000000000056efbf in start_vhost_loop (dummy=0x0) at lib/netdev-
> dpdk.c:1789
> #5  0x00000000004f60f3 in ovsthread_wrapper (aux_=0x25248a0) at lib/ovs-
> thread.c:340
> #6  0x00007fca43a1edf5 in start_thread (arg=0x7fca3c13d700) at
> pthread_create.c:308
> #7  0x00007fca432421ad in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> (gdb)
> 
> 

I can reproduce a segfault when using QEMU v2.4.0+ like you mentioned you are. It looks like something has gone in between v2.3.0 and v2.4.0 in QEMU that is causing this issue.
Can you please try your test with the QEMU version we recommend to use in INSTALL.DPDK which is v2.2.0.

Thanks,
Ciara

> 
> 
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00021|dpdk|INFO|Port 0: a0:36:9f:5a:cb:1c
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00007|dpif_netdev(pmd16)|INFO|Core 0 processing port 'vhost-user1'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00008|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk0'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00009|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk1'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00010|dpif_netdev(pmd16)|INFO|Core 0 processing port 'vhost-user2'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00022|bridge|INFO|bridge br0: added interface dpdk0 on port 2
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00023|bridge|INFO|bridge br0: using datapath ID 0000a0369f5acb1c
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00024|connmgr|INFO|br0: added service controller
> "punix:/usr/local/var/run/openvswitch/br0.mgmt"
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00025|dpif_netdev|INFO|Created 1 pmd threads on numa node 0
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00026|timeval|WARN|Unreasonably long 2481ms poll interval (1726ms
> user, 2ms system)
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00027|timeval|WARN|faults: 2032 minor, 0 major
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00028|timeval|WARN|disk: 0 reads, 8 writes
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00029|timeval|WARN|context switches: 143 voluntary, 426 involuntary
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00030|coverage|INFO|Event coverage, avg rate over last: 5 seconds,
> last minute, last hour,  hash=66963b7f:
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00031|coverage|INFO|bridge_reconfigure         0.0/sec     0.000/sec        0.
> 0000/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00032|coverage|INFO|ofproto_flush              0.0/sec     0.000/sec        0.00
> 00/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00033|coverage|INFO|ofproto_update_port        0.0/sec     0.000/sec
> 0.0000/sec   total: 5
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00034|coverage|INFO|rev_flow_table             0.0/sec     0.000/sec        0.00
> 00/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00035|coverage|INFO|cmap_expand                0.0/sec     0.000/sec        0.0
> 000/sec   total: 3
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00036|coverage|INFO|cmap_shrink                0.0/sec     0.000/sec        0.000
> 0/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00037|coverage|INFO|dpif_port_add              0.0/sec     0.000/sec        0.00
> 00/sec   total: 5
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00038|coverage|INFO|dpif_flow_flush            0.0/sec     0.000/sec        0.00
> 00/sec   total: 2
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00039|coverage|INFO|dpif_flow_get              0.0/sec     0.000/sec        0.00
> 00/sec   total: 5
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00040|coverage|INFO|dpif_flow_put              0.0/sec     0.000/sec        0.00
> 00/sec   total: 5
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00041|coverage|INFO|dpif_flow_del              0.0/sec     0.000/sec        0.00
> 00/sec   total: 5
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00042|coverage|INFO|dpif_execute               0.0/sec     0.000/sec        0.00
> 00/sec   total: 2
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00043|coverage|INFO|flow_extract               0.0/sec     0.000/sec        0.000
> 0/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00044|coverage|INFO|hmap_pathological          0.0/sec     0.000/sec        0.
> 0000/sec   total: 2
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00001|dpif_netdev(pmd24)|INFO|Core 1 processing port 'vhost-user1'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00045|coverage|INFO|hmap_expand                0.0/sec     0.000/sec        0.0
> 000/sec   total: 488
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00046|coverage|INFO|netdev_get_stats           0.0/sec     0.000/sec        0.
> 0000/sec   total: 10
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00002|dpif_netdev(pmd24)|INFO|Core 1 processing port 'dpdk0'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00047|coverage|INFO|txn_unchanged              0.0/sec     0.000/sec        0.0
> 000/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00003|dpif_netdev(pmd24)|INFO|Core 1 processing port 'dpdk1'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00048|coverage|INFO|txn_incomplete             0.0/sec     0.000/sec        0.0
> 000/sec   total: 2
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00004|dpif_netdev(pmd24)|INFO|Core 1 processing port 'vhost-user2'
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00049|coverage|INFO|poll_create_node           0.0/sec     0.000/sec        0.
> 0000/sec   total: 229
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00050|coverage|INFO|poll_zero_timeout          0.0/sec     0.000/sec        0.
> 0000/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00051|coverage|INFO|pstream_open               0.0/sec     0.000/sec        0.0
> 000/sec   total: 3
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00052|coverage|INFO|stream_open                0.0/sec     0.000/sec        0.00
> 00/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00053|coverage|INFO|util_xalloc                0.0/sec     0.000/sec        0.0000/
> sec   total: 13886
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00054|coverage|INFO|netdev_set_policing        0.0/sec     0.000/sec        0
> .0000/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00055|coverage|INFO|netdev_get_ifindex         0.0/sec     0.000/sec        0
> .0000/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00056|coverage|INFO|netdev_get_hwaddr          0.0/sec     0.000/sec
> 0.0000/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00057|coverage|INFO|netdev_set_hwaddr          0.0/sec     0.000/sec
> 0.0000/sec   total: 1
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00058|coverage|INFO|netdev_get_ethtool         0.0/sec     0.000/sec
> 0.0000/sec   total: 2
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00059|coverage|INFO|netlink_received           0.0/sec     0.000/sec        0.0
> 000/sec   total: 23
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00060|coverage|INFO|netlink_recv_jumbo         0.0/sec     0.000/sec        0
> .0000/sec   total: 2
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00061|coverage|INFO|netlink_sent               0.0/sec     0.000/sec        0.000
> 0/sec   total: 12
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00062|coverage|INFO|nln_changed                0.0/sec     0.000/sec        0.00
> 00/sec   total: 5
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00063|coverage|INFO|67 events never hit
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00064|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.4.0
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00065|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00066|memory|INFO|8252 kB peak resident set size after 10.5 seconds
> Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00067|memory|INFO|handlers:8 ports:5 revalidators:4 rules:5
> Oct 09 15:53:51 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00068|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> Oct 09 15:53:51 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00069|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> Oct 09 15:53:52 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00070|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> Oct 09 15:53:52 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00071|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00072|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00073|poll_loop|INFO|wakeup due to 0-ms timeout at
> ofproto/ofproto-dpif.c:1571 (57% CPU usage)
> Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00074|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/ovs-rcu.c:206 (57% CPU usage)
> Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00075|poll_loop|INFO|wakeup due to [POLLIN] on fd 79 (<-
> >/usr/local/var/run/openvswitch/db.sock) at l...PU usage)
> Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> ovs|00076|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> new virtio connection is 110
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> new device, handle is 0
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_OWNER
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_GET_FEATURES
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_CALL
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> vring call idx:0 file:111
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_CALL
> Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> vring call idx:1 file:112
> Oct 09 15:54:29 localhost.localdomain kvm[22204]: 1 guest now active
> Oct 09 15:54:39 localhost.localdomain kernel: kvm [22199]: vcpu0 disabled
> perfctr wrmsr: 0xc1 data 0xffff
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_CALL
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> vring call idx:0 file:113
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_CALL
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> vring call idx:1 file:111
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_FEATURES
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_MEM_TABLE
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> mapped region 0 fd:112 to 0xffffffffffffffff sz:0xffffab4a9f800000
> off:0xffffab4a5f800000
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> mmap qemu guest failed.
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_NUM
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_BASE
> Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_ADDR
> Oct 09 15:54:40 localhost.localdomain kernel: vhost_thread13[22047]:
> segfault at 18 ip 00007fbb9ffc5f2c sp 00007fbb9822b670 error 4 in
> libdpdk.so[7fbb9ff26000+1b3000]
> Oct 09 15:54:41 localhost.localdomain abrt-hook-ccpp[22244]: Saved core
> dump of pid 22045 (/root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-
> vswitchd) to /var/spool/...08 bytes)
> 
> On Fri, Oct 9, 2015 at 12:07 AM, Loftus, Ciara <ciara.loftus at intel.com> wrote:
> >
> > I have ran into a ovs-vswitchd SEGV while trying to use the vhost-user
> > feature with the latest dpdk. I have attempted to provide as much detail as
> I
> > can.
> >
> > versions compiled from source on CentOS 7.1 x86_64:
> >  - openvswitch 2.4.0 - http://openvswitch.org/releases/openvswitch-
> > 2.4.0.tar.gz
> >  - dpdk 2.1.0 - http://dpdk.org/browse/dpdk/snapshot/dpdk-2.1.0.tar.gz
> >  - qemu - 2.4.0.1 - http://wiki.qemu-project.org/download/qemu-
> > 2.4.0.1.tar.bz2
> >
> >
> > Steps to reproduce:
> >
> > - Build and Install following these instructions
> > http://wiki.qemu.org/Features/vhost-user-ovs-dpdk
> > - setup details:
> >
> > hugepage config:
> > GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap
> > crashkernel=auto rhgb quiet iommu=pt intel_iommu=on
> > default_hugepagesz=1G hugepagesz=1G hugepages=8
> > isolcpus=1,2,3,4,5,12,13,14,15,16,17,7,8,9,10,11,18,19,20,21,22,23
> > transparent_hugepage=never"
> >
> > # export DPDK_DIR=/root/dpdk/dpdk-2.1.0
> > # export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/
> > # export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> >
> >
> > # modprobe openvswitch
> > # modprobe uio
> > # insmod /root/dpdk/dpdk-2.1.0/x86_64-native-linuxapp-
> > gcc/kmod/igb_uio.ko
> > # insmod /root/dpdk/dpdk-
> > 2.1.0/lib/librte_vhost/eventfd_link/eventfd_link.ko
> >
> > # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f0
> > # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f1
> >
> > # /root/openvswitch/openvswitch-2.4.0/ovsdb/ovsdb-server --
> > remote=punix:/usr/local/var/run/openvswitch/db.sock     --
> > remote=db:Open_vSwitch,Open_vSwitch,manager_options     --pidfile --
> > detach
> >
> > # rm -f /usr/local/var/run/openvswitch/vhost-user*
> >
> > # /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-vswitchd --dpdk -c
> > 0xffff -n 4 --socket-mem 1024 -- unix:$DB_SOCK --pidfile --detach --log-
> > file=/var/log/openvswitch/ovs-vswitchd.log
> >
> > # lspci | grep Ethernet
> > 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit
> > X540-AT2 (rev 01)
> > 01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit
> > X540-AT2 (rev 01)
> >
> > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-br br0 -- set
> > bridge br0 datapath_type=netdev
> > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > dpdk0 -- set Interface dpdk0 type=dpdk
> > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > dpdk1 -- set Interface dpdk1 type=dpdk
> > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser
> > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
> >
> > - Start the first VM like this:
> > ./qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 \
> 
> My first suggestion would be to try supplying your VM with more memory.
> ie. instead of "-m 1024", try "-m 4096"
> 
> Thanks,
> Ciara
> 
> >  -display sdl \
> >  -sdl \
> >  -chardev socket,id=char0,path=/usr/local/var/run/openvswitch/vhost-
> user1
> > \
> >  -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce  \
> >  -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01 \
> >  -object memory-backend-file,id=mem,size=1024M,mem-
> > path=/dev/hugepages,share=on \
> >  -numa node,memdev=mem -mem-prealloc \
> >  -net user,hostfwd=tcp::10021-:22 \
> >  -net nic  \
> >  /var/lib/libvirt/images/centos-vm1.qcow2  &
> >
> > - From another machine send iperf traffic to this VM.
> >   iperf -s (in the VM)
> >   iperf -c <ipaddr> -t 60 -P 12 (from another machine)
> >
> > - After about a few seconds, iperf stalls. network connectivity breaks. Can't
> > ping the VM anymore.
> > - Saw these in the ovs-vswitchd log files.
> >
> > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > Failed to allocate memory for mbuf.
> > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > Failed to allocate memory for mbuf.
> > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > Failed to allocate memory for mbuf.
> > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > Failed to allocate memory for mbuf.
> > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > Failed to allocate memory for mbuf.
> >
> > - shutdown iperf ... then shutdown the VM.
> > - at this point ovs-vswitchd crashes with the following SEGV.
> >
> > [root at localhost vswitchd]# gdb ./ovs-vswitchd /var/spool/abrt/ccpp-2015-
> > 09-29-18\:47\:37-22012/coredump
> > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
> > Copyright (C) 2013 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later
> > <http://gnu.org/licenses/gpl.html>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.  Type "show
> > copying"
> > and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>...
> > Reading symbols from /root/openvswitch/openvswitch-
> 2.4.0/vswitchd/ovs-
> > vswitchd...done.
> > [New LWP 22051]
> > [New LWP 22012]
> > [New LWP 22013]
> > [New LWP 22015]
> > [New LWP 22039]
> > [New LWP 22040]
> > [New LWP 22041]
> > [New LWP 22042]
> > [New LWP 22043]
> > [New LWP 22044]
> > [New LWP 22045]
> > [New LWP 22046]
> > [New LWP 22047]
> > [New LWP 22048]
> > [New LWP 22049]
> > [New LWP 22050]
> > [New LWP 22014]
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `/root/openvswitch/openvswitch-
> > 2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> > /lib64/libdpdk.so
> > Missing separate debuginfos, use: debuginfo-install glibc-2.17-
> 78.el7.x86_64
> > (gdb) where
> > #0  0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> > /lib64/libdpdk.so
> > #1  0x000000000056d103 in netdev_dpdk_vhost_rxq_recv
> > (rxq_=0x7efe5a537600, packets=0x7eff4c9a2930, c=0x7eff4c9a292c)
> >     at lib/netdev-dpdk.c:952
> > #2  0x00000000004a2dec in netdev_rxq_recv (rx=0x7efe5a537600,
> > buffers=0x7eff4c9a2930, cnt=0x7eff4c9a292c)
> >     at lib/netdev.c:652
> > #3  0x0000000000478f33 in dp_netdev_process_rxq_port (pmd=0x231c310,
> > port=0x2166990, rxq=0x7efe5a537600)
> >     at lib/dpif-netdev.c:2520
> > #4  0x0000000000479648 in pmd_thread_main (f_=0x231c310) at lib/dpif-
> > netdev.c:2680
> > #5  0x00000000004f60f3 in ovsthread_wrapper (aux_=0x231c150) at lib/ovs-
> > thread.c:340
> > #6  0x00007eff4d276df5 in start_thread () from /lib64/libpthread.so.0
> > #7  0x00007eff4ca9a1ad in clone () from /lib64/libc.so.6



More information about the discuss mailing list