[ovs-discuss] ovs-vswitchd SEGV when using vhost-user feature with dpdk

Loftus, Ciara ciara.loftus at intel.com
Wed Oct 28 11:38:31 UTC 2015


> 
> Thanks for the response. I switched to QEMU - 2.2.0. No SEGV this time
> around. But even with -m 4096, I get a lot of these messages and the VM
> throws a kernel trace - like I reported on my first message.

Sorry for the delay in response. Some things to try to overcome your issue with v2.2.0:
- Increase the number of hugepages you allocate at boot:
Eg. default_hugepagesz=1G hugepagesz=1G hugepages=16
- Increase the ovs-vswitchd socket-mem:
Eg. --socket-mem 2048,0
- Once again, try allocating more mem to QEMU
Eg. -m 4096M

These are the figures I would typically use so hopefully they work for you too.

Thanks,
Ciara

> 
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_DATA:
> Failed to allocate memory for mbuf.
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_CONFIG:
> read message VHOST_USER_SET_VRING_CALL
> Oct 12 10:43:49 localhost.localdomain ovs-vswitchd[29441]: VHOST_CONFIG:
> vring call idx:0 file:111
>                                                            VHOST_DATA: Failed to allocate memory for
> mbuf.
> 
> 
> On Mon, Oct 12, 2015 at 3:49 AM, Loftus, Ciara <ciara.loftus at intel.com>
> wrote:
> >
> > Increasing qemu to -m 4096, causes an immediate ovs-vswitchd SEGV
> >
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `/root/openvswitch/openvswitch-
> > 2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x00007fca43ed6f2c in set_vring_addr () from /lib64/libdpdk.so
> > (gdb) where
> > #0  0x00007fca43ed6f2c in set_vring_addr () from /lib64/libdpdk.so
> > #1  0x00007fca43f90d2e in vserver_message_handler () from
> > /lib64/libdpdk.so
> > #2  0x00007fca43f3ad7d in fdset_event_dispatch () from /lib64/libdpdk.so
> > #3  0x00007fca43f91330 in rte_vhost_driver_session_start () from
> > /lib64/libdpdk.so
> > #4  0x000000000056efbf in start_vhost_loop (dummy=0x0) at lib/netdev-
> > dpdk.c:1789
> > #5  0x00000000004f60f3 in ovsthread_wrapper (aux_=0x25248a0) at
> lib/ovs-
> > thread.c:340
> > #6  0x00007fca43a1edf5 in start_thread (arg=0x7fca3c13d700) at
> > pthread_create.c:308
> > #7  0x00007fca432421ad in clone () at
> > ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
> > (gdb)
> >
> >
> 
> I can reproduce a segfault when using QEMU v2.4.0+ like you mentioned you
> are. It looks like something has gone in between v2.3.0 and v2.4.0 in QEMU
> that is causing this issue.
> Can you please try your test with the QEMU version we recommend to use in
> INSTALL.DPDK which is v2.2.0.
> 
> Thanks,
> Ciara
> 
> >
> >
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00021|dpdk|INFO|Port 0: a0:36:9f:5a:cb:1c
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00007|dpif_netdev(pmd16)|INFO|Core 0 processing port 'vhost-
> user1'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00008|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk0'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00009|dpif_netdev(pmd16)|INFO|Core 0 processing port 'dpdk1'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00010|dpif_netdev(pmd16)|INFO|Core 0 processing port 'vhost-
> user2'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00022|bridge|INFO|bridge br0: added interface dpdk0 on port 2
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00023|bridge|INFO|bridge br0: using datapath ID 0000a0369f5acb1c
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00024|connmgr|INFO|br0: added service controller
> > "punix:/usr/local/var/run/openvswitch/br0.mgmt"
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00025|dpif_netdev|INFO|Created 1 pmd threads on numa node 0
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00026|timeval|WARN|Unreasonably long 2481ms poll interval
> (1726ms
> > user, 2ms system)
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00027|timeval|WARN|faults: 2032 minor, 0 major
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00028|timeval|WARN|disk: 0 reads, 8 writes
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00029|timeval|WARN|context switches: 143 voluntary, 426
> involuntary
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00030|coverage|INFO|Event coverage, avg rate over last: 5 seconds,
> > last minute, last hour,  hash=66963b7f:
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00031|coverage|INFO|bridge_reconfigure         0.0/sec     0.000/sec        0.
> > 0000/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00032|coverage|INFO|ofproto_flush              0.0/sec     0.000/sec        0.00
> > 00/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00033|coverage|INFO|ofproto_update_port        0.0/sec     0.000/sec
> > 0.0000/sec   total: 5
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00034|coverage|INFO|rev_flow_table             0.0/sec     0.000/sec        0.00
> > 00/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00035|coverage|INFO|cmap_expand                0.0/sec     0.000/sec        0.0
> > 000/sec   total: 3
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00036|coverage|INFO|cmap_shrink                0.0/sec     0.000/sec        0.000
> > 0/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00037|coverage|INFO|dpif_port_add              0.0/sec     0.000/sec        0.00
> > 00/sec   total: 5
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00038|coverage|INFO|dpif_flow_flush            0.0/sec     0.000/sec        0.00
> > 00/sec   total: 2
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00039|coverage|INFO|dpif_flow_get              0.0/sec     0.000/sec        0.00
> > 00/sec   total: 5
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00040|coverage|INFO|dpif_flow_put              0.0/sec     0.000/sec        0.00
> > 00/sec   total: 5
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00041|coverage|INFO|dpif_flow_del              0.0/sec     0.000/sec        0.00
> > 00/sec   total: 5
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00042|coverage|INFO|dpif_execute               0.0/sec     0.000/sec        0.00
> > 00/sec   total: 2
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00043|coverage|INFO|flow_extract               0.0/sec     0.000/sec        0.000
> > 0/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00044|coverage|INFO|hmap_pathological          0.0/sec     0.000/sec        0.
> > 0000/sec   total: 2
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00001|dpif_netdev(pmd24)|INFO|Core 1 processing port 'vhost-
> user1'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00045|coverage|INFO|hmap_expand                0.0/sec     0.000/sec        0.0
> > 000/sec   total: 488
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00046|coverage|INFO|netdev_get_stats           0.0/sec     0.000/sec        0.
> > 0000/sec   total: 10
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00002|dpif_netdev(pmd24)|INFO|Core 1 processing port 'dpdk0'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00047|coverage|INFO|txn_unchanged              0.0/sec     0.000/sec        0.0
> > 000/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00003|dpif_netdev(pmd24)|INFO|Core 1 processing port 'dpdk1'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00048|coverage|INFO|txn_incomplete             0.0/sec     0.000/sec        0.0
> > 000/sec   total: 2
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00004|dpif_netdev(pmd24)|INFO|Core 1 processing port 'vhost-
> user2'
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00049|coverage|INFO|poll_create_node           0.0/sec     0.000/sec        0.
> > 0000/sec   total: 229
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00050|coverage|INFO|poll_zero_timeout          0.0/sec     0.000/sec        0.
> > 0000/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00051|coverage|INFO|pstream_open               0.0/sec     0.000/sec        0.0
> > 000/sec   total: 3
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00052|coverage|INFO|stream_open                0.0/sec     0.000/sec        0.00
> > 00/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00053|coverage|INFO|util_xalloc                0.0/sec     0.000/sec        0.0000/
> > sec   total: 13886
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00054|coverage|INFO|netdev_set_policing        0.0/sec     0.000/sec        0
> > .0000/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00055|coverage|INFO|netdev_get_ifindex         0.0/sec     0.000/sec        0
> > .0000/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00056|coverage|INFO|netdev_get_hwaddr          0.0/sec     0.000/sec
> > 0.0000/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00057|coverage|INFO|netdev_set_hwaddr          0.0/sec     0.000/sec
> > 0.0000/sec   total: 1
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00058|coverage|INFO|netdev_get_ethtool         0.0/sec     0.000/sec
> > 0.0000/sec   total: 2
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00059|coverage|INFO|netlink_received           0.0/sec     0.000/sec        0.0
> > 000/sec   total: 23
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00060|coverage|INFO|netlink_recv_jumbo         0.0/sec     0.000/sec        0
> > .0000/sec   total: 2
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00061|coverage|INFO|netlink_sent               0.0/sec     0.000/sec        0.000
> > 0/sec   total: 12
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> >
> ovs|00062|coverage|INFO|nln_changed                0.0/sec     0.000/sec        0.00
> > 00/sec   total: 5
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00063|coverage|INFO|67 events never hit
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00064|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.4.0
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00065|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00066|memory|INFO|8252 kB peak resident set size after 10.5
> seconds
> > Oct 09 15:53:50 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00067|memory|INFO|handlers:8 ports:5 revalidators:4 rules:5
> > Oct 09 15:53:51 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00068|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> > Oct 09 15:53:51 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00069|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> > Oct 09 15:53:52 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00070|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> > Oct 09 15:53:52 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00071|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> > Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00072|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> > Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00073|poll_loop|INFO|wakeup due to 0-ms timeout at
> > ofproto/ofproto-dpif.c:1571 (57% CPU usage)
> > Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00074|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/ovs-rcu.c:206 (57% CPU usage)
> > Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00075|poll_loop|INFO|wakeup due to [POLLIN] on fd 79 (<-
> > >/usr/local/var/run/openvswitch/db.sock) at l...PU usage)
> > Oct 09 15:53:53 localhost.localdomain ovs-vswitchd[22045]:
> > ovs|00076|poll_loop|INFO|wakeup due to [POLLIN] on fd 95 (FIFO
> > pipe:[12835268]) at lib/dpif-netdev.c:2592...PU usage)
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > new virtio connection is 110
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > new device, handle is 0
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_OWNER
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_GET_FEATURES
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_VRING_CALL
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > vring call idx:0 file:111
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_VRING_CALL
> > Oct 09 15:54:29 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > vring call idx:1 file:112
> > Oct 09 15:54:29 localhost.localdomain kvm[22204]: 1 guest now active
> > Oct 09 15:54:39 localhost.localdomain kernel: kvm [22199]: vcpu0 disabled
> > perfctr wrmsr: 0xc1 data 0xffff
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_VRING_CALL
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > vring call idx:0 file:113
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_VRING_CALL
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > vring call idx:1 file:111
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_FEATURES
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_MEM_TABLE
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > mapped region 0 fd:112 to 0xffffffffffffffff sz:0xffffab4a9f800000
> > off:0xffffab4a5f800000
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > mmap qemu guest failed.
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_VRING_NUM
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_VRING_BASE
> > Oct 09 15:54:40 localhost.localdomain ovs-vswitchd[22045]:
> VHOST_CONFIG:
> > read message VHOST_USER_SET_VRING_ADDR
> > Oct 09 15:54:40 localhost.localdomain kernel: vhost_thread13[22047]:
> > segfault at 18 ip 00007fbb9ffc5f2c sp 00007fbb9822b670 error 4 in
> > libdpdk.so[7fbb9ff26000+1b3000]
> > Oct 09 15:54:41 localhost.localdomain abrt-hook-ccpp[22244]: Saved core
> > dump of pid 22045 (/root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-
> > vswitchd) to /var/spool/...08 bytes)
> >
> > On Fri, Oct 9, 2015 at 12:07 AM, Loftus, Ciara <ciara.loftus at intel.com>
> wrote:
> > >
> > > I have ran into a ovs-vswitchd SEGV while trying to use the vhost-user
> > > feature with the latest dpdk. I have attempted to provide as much detail
> as
> > I
> > > can.
> > >
> > > versions compiled from source on CentOS 7.1 x86_64:
> > >  - openvswitch 2.4.0 - http://openvswitch.org/releases/openvswitch-
> > > 2.4.0.tar.gz
> > >  - dpdk 2.1.0 - http://dpdk.org/browse/dpdk/snapshot/dpdk-2.1.0.tar.gz
> > >  - qemu - 2.4.0.1 - http://wiki.qemu-project.org/download/qemu-
> > > 2.4.0.1.tar.bz2
> > >
> > >
> > > Steps to reproduce:
> > >
> > > - Build and Install following these instructions
> > > http://wiki.qemu.org/Features/vhost-user-ovs-dpdk
> > > - setup details:
> > >
> > > hugepage config:
> > > GRUB_CMDLINE_LINUX="rd.lvm.lv=centos/root rd.lvm.lv=centos/swap
> > > crashkernel=auto rhgb quiet iommu=pt intel_iommu=on
> > > default_hugepagesz=1G hugepagesz=1G hugepages=8
> > > isolcpus=1,2,3,4,5,12,13,14,15,16,17,7,8,9,10,11,18,19,20,21,22,23
> > > transparent_hugepage=never"
> > >
> > > # export DPDK_DIR=/root/dpdk/dpdk-2.1.0
> > > # export DPDK_BUILD=$DPDK_DIR/x86_64-native-linuxapp-gcc/
> > > # export DB_SOCK=/usr/local/var/run/openvswitch/db.sock
> > >
> > >
> > > # modprobe openvswitch
> > > # modprobe uio
> > > # insmod /root/dpdk/dpdk-2.1.0/x86_64-native-linuxapp-
> > > gcc/kmod/igb_uio.ko
> > > # insmod /root/dpdk/dpdk-
> > > 2.1.0/lib/librte_vhost/eventfd_link/eventfd_link.ko
> > >
> > > # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f0
> > > # /root/dpdk/dpdk-2.1.0/tools/dpdk_nic_bind.py --bind=igb_uio ens1f1
> > >
> > > # /root/openvswitch/openvswitch-2.4.0/ovsdb/ovsdb-server --
> > > remote=punix:/usr/local/var/run/openvswitch/db.sock     --
> > > remote=db:Open_vSwitch,Open_vSwitch,manager_options     --pidfile --
> > > detach
> > >
> > > # rm -f /usr/local/var/run/openvswitch/vhost-user*
> > >
> > > # /root/openvswitch/openvswitch-2.4.0/vswitchd/ovs-vswitchd --dpdk -
> c
> > > 0xffff -n 4 --socket-mem 1024 -- unix:$DB_SOCK --pidfile --detach --log-
> > > file=/var/log/openvswitch/ovs-vswitchd.log
> > >
> > > # lspci | grep Ethernet
> > > 01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-
> Gigabit
> > > X540-AT2 (rev 01)
> > > 01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-
> Gigabit
> > > X540-AT2 (rev 01)
> > >
> > > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-br br0 --
> set
> > > bridge br0 datapath_type=netdev
> > > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > > dpdk0 -- set Interface dpdk0 type=dpdk
> > > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > > dpdk1 -- set Interface dpdk1 type=dpdk
> > > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > > vhost-user2 -- set Interface vhost-user2 type=dpdkvhostuser
> > > # /root/openvswitch/openvswitch-2.4.0/utilities/ovs-vsctl add-port br0
> > > vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser
> > >
> > > - Start the first VM like this:
> > > ./qemu-system-x86_64 -enable-kvm -m 1024 -smp 2 \
> >
> > My first suggestion would be to try supplying your VM with more memory.
> > ie. instead of "-m 1024", try "-m 4096"
> >
> > Thanks,
> > Ciara
> >
> > >  -display sdl \
> > >  -sdl \
> > >  -chardev socket,id=char0,path=/usr/local/var/run/openvswitch/vhost-
> > user1
> > > \
> > >  -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce  \
> > >  -device virtio-net-pci,netdev=mynet1,mac=52:54:00:02:d9:01 \
> > >  -object memory-backend-file,id=mem,size=1024M,mem-
> > > path=/dev/hugepages,share=on \
> > >  -numa node,memdev=mem -mem-prealloc \
> > >  -net user,hostfwd=tcp::10021-:22 \
> > >  -net nic  \
> > >  /var/lib/libvirt/images/centos-vm1.qcow2  &
> > >
> > > - From another machine send iperf traffic to this VM.
> > >   iperf -s (in the VM)
> > >   iperf -c <ipaddr> -t 60 -P 12 (from another machine)
> > >
> > > - After about a few seconds, iperf stalls. network connectivity breaks.
> Can't
> > > ping the VM anymore.
> > > - Saw these in the ovs-vswitchd log files.
> > >
> > > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > > Failed to allocate memory for mbuf.
> > > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > > Failed to allocate memory for mbuf.
> > > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > > Failed to allocate memory for mbuf.
> > > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > > Failed to allocate memory for mbuf.
> > > Sep 29 17:51:08 localhost.localdomain ovs-vswitchd[6483]: VHOST_DATA:
> > > Failed to allocate memory for mbuf.
> > >
> > > - shutdown iperf ... then shutdown the VM.
> > > - at this point ovs-vswitchd crashes with the following SEGV.
> > >
> > > [root at localhost vswitchd]# gdb ./ovs-vswitchd /var/spool/abrt/ccpp-
> 2015-
> > > 09-29-18\:47\:37-22012/coredump
> > > GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-64.el7
> > > Copyright (C) 2013 Free Software Foundation, Inc.
> > > License GPLv3+: GNU GPL version 3 or later
> > > <http://gnu.org/licenses/gpl.html>
> > > This is free software: you are free to change and redistribute it.
> > > There is NO WARRANTY, to the extent permitted by law.  Type "show
> > > copying"
> > > and "show warranty" for details.
> > > This GDB was configured as "x86_64-redhat-linux-gnu".
> > > For bug reporting instructions, please see:
> > > <http://www.gnu.org/software/gdb/bugs/>...
> > > Reading symbols from /root/openvswitch/openvswitch-
> > 2.4.0/vswitchd/ovs-
> > > vswitchd...done.
> > > [New LWP 22051]
> > > [New LWP 22012]
> > > [New LWP 22013]
> > > [New LWP 22015]
> > > [New LWP 22039]
> > > [New LWP 22040]
> > > [New LWP 22041]
> > > [New LWP 22042]
> > > [New LWP 22043]
> > > [New LWP 22044]
> > > [New LWP 22045]
> > > [New LWP 22046]
> > > [New LWP 22047]
> > > [New LWP 22048]
> > > [New LWP 22049]
> > > [New LWP 22050]
> > > [New LWP 22014]
> > > [Thread debugging using libthread_db enabled]
> > > Using host libthread_db library "/lib64/libthread_db.so.1".
> > > Core was generated by `/root/openvswitch/openvswitch-
> > > 2.4.0/vswitchd/ovs-vswitchd --dpdk -c 0xffff -n 4'.
> > > Program terminated with signal 11, Segmentation fault.
> > > #0  0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> > > /lib64/libdpdk.so
> > > Missing separate debuginfos, use: debuginfo-install glibc-2.17-
> > 78.el7.x86_64
> > > (gdb) where
> > > #0  0x00007eff4d71df1f in rte_vhost_dequeue_burst () from
> > > /lib64/libdpdk.so
> > > #1  0x000000000056d103 in netdev_dpdk_vhost_rxq_recv
> > > (rxq_=0x7efe5a537600, packets=0x7eff4c9a2930, c=0x7eff4c9a292c)
> > >     at lib/netdev-dpdk.c:952
> > > #2  0x00000000004a2dec in netdev_rxq_recv (rx=0x7efe5a537600,
> > > buffers=0x7eff4c9a2930, cnt=0x7eff4c9a292c)
> > >     at lib/netdev.c:652
> > > #3  0x0000000000478f33 in dp_netdev_process_rxq_port
> (pmd=0x231c310,
> > > port=0x2166990, rxq=0x7efe5a537600)
> > >     at lib/dpif-netdev.c:2520
> > > #4  0x0000000000479648 in pmd_thread_main (f_=0x231c310) at lib/dpif-
> > > netdev.c:2680
> > > #5  0x00000000004f60f3 in ovsthread_wrapper (aux_=0x231c150) at
> lib/ovs-
> > > thread.c:340
> > > #6  0x00007eff4d276df5 in start_thread () from /lib64/libpthread.so.0
> > > #7  0x00007eff4ca9a1ad in clone () from /lib64/libc.so.6



More information about the discuss mailing list