[ovs-discuss] ovs-vswitchd crash

[MAO Ruoyu]

Hi，
Our host kernel is " Linux host16 3.17.4-301.fc21.x86_64 #1 SMP Thu Nov 27 19:09:10 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux "
Our guest kernel is
" Red Hat Enterprise Linux Server 7.1(Maipo) Kernel 3.10.0-229.20.1.e17.x86_64 on an x86_64 "

We try to use DPDK2.1. but no effect.





-----Original Message-----
From: Traynor, Kevin [mailto:kevin.traynor at intel.com]
Sent: 2015年12月11日 23:49
To: MAO Ruoyu; discuss at openvswitch.org
Cc: dev at dpdk.org
Subject: RE: ovs-vswitchd crash


> -----Original Message-----
> From: discuss [mailto:discuss-bounces at openvswitch.org] On Behalf Of 
> MAO Ruoyu
> Sent: Friday, December 11, 2015 2:25 AM
> To: discuss at openvswitch.org
> Subject: [ovs-discuss] ovs-vswitchd crash
> 
> Hi,
> We install openstack using devstack with OVS-DPDK We use host8 as 
> compute node, use host16 as controller node. Now we create vms on both 
> of them as bellow.
> on host8, we create vms V7510_OVS_DPDK-SCM11 and V7510_OVS_DPDK-PIM20; 
> on host16, we create vms V7510_OVS_DPDK-SCM10 and V7510_OVS_DPDK-PIM19.
> we create vms successfully.
> But after reboot of  V7510_OVS_DPDK-SCM11 application, the 
> ovs-vswitchd on
> host8 crashed.
> We don't know whether the reboot leads the service crashed or the 
> service crash leads the reboot.

Cross posting to dpdk-dev.

What kernel version are you running in the guest? there was previously an issue with the DPDK vhost-library and 3.19 I think.

It might be worth to test OVS head of master with DPDK 2.1, or DPDK 2.2-rc3.

> 
> We try QEMU 2.3.1, QEMU 2.2.1 and QEMU 2.1.3 on host8, but no effect.
> Now we use DPDK2.0, OVS2.4.
> 
> [root at host8 ~]# coredumpctl gdb 5295
>            PID: 5295 (ovs-vswitchd)
>            UID: 0 (root)
>            GID: 107 (qemu)
>         Signal: 11 (SEGV)
>      Timestamp: Thu 2015-12-10 16:49:17 CST (16h ago)
>   Command Line: /usr/sbin/ovs-vswitchd --dpdk -vhost_sock_dir 
> /var/run/openvswitch -c 2 -n 4 --proc-type primary --huge-dir 
> /mnt/huge -- socket-mem 2048 2048 -- unix:/var/run/openvswitch/db.sock
>     Executable: /usr/sbin/ovs-vswitchd Control Group: 
> /user.slice/user-1000.slice/session-1.scope
>           Unit: session-1.scope
>          Slice: user-1000.slice
>        Session: 1
>      Owner UID: 1000 (diag)
>        Boot ID: beb01a89b99f494f858c529df3cbf6a2
>     Machine ID: 793afa475dbd4b84949ba36a4a0fcd13
>       Hostname: host8
>       Coredump: /var/lib/systemd/coredump/core.ovs-
> vswitchd.0.beb01a89b99f494f858c529df3cbf6a2.5295.1449737357000000.xz
>        Message: Process 5295 (ovs-vswitchd) of user 0 dumped core.
> 
>                 Stack trace of thread 5601:
>                 #0  0x0000000000530852 rte_log_add_in_history 
> (ovs-vswitchd)
>                 #1  0x00000000004c9cbe console_log_write 
> (ovs-vswitchd)
>                 #2  0x00007fcd9c9d9501 _IO_cookie_write (libc.so.6)
>                 #3  0x00007fcd9c9e51d9 _IO_do_write@@GLIBC_2.2.5 
> (libc.so.6)
>                 #4  0x00007fcd9c9e35d0 _IO_file_sync@@GLIBC_2.2.5 
> (libc.so.6)
>                 #5  0x00007fcd9c9d8f66 _IO_fflush (libc.so.6)
>                 #6  0x00000000005315f7 rte_vlog (ovs-vswitchd)
>                #7  0x000000000040648d rte_log (ovs-vswitchd)
>                 #8  0x000000000046c930 rte_vhost_enqueue_burst 
> (ovs-vswitchd)
>                 #9  0x0000000000676051 __netdev_dpdk_vhost_send (ovs-
> vswitchd)
>                 #10 0x0000000000676967 netdev_dpdk_vhost_send 
> (ovs-vswitchd)
>                 #11 0x00000000005bcda0 netdev_send (ovs-vswitchd)
>                 #12 0x0000000000599d21 dp_execute_cb (ovs-vswitchd)
>                 #13 0x00000000005c60f4 odp_execute_actions 
> (ovs-vswitchd)
>                 #14 0x0000000000598e6c dp_netdev_input (ovs-vswitchd)
>                 #15 0x0000000000599197 dp_netdev_process_rxq_port 
> (ovs-
> vswitchd)
>                 #16 0x000000000059981c pmd_thread_main (ovs-vswitchd)
>                 #17 0x000000000060ba34 ovsthread_wrapper 
> (ovs-vswitchd)
>                 #18 0x00007fcd9d47d52a start_thread (libpthread.so.0)
>                 #19 0x00007fcd9ca6b22d __clone (libc.so.6)
> 
>                 Stack trace of thread 5600:
>                 #0  0x000000000046d675 rte_vhost_dequeue_burst 
> (ovs-vswitchd)
>                 #1  0x0000000000672bbc netdev_dpdk_vhost_rxq_recv 
> (ovs-
> vswitchd)
>                 #2  0x00000000005bcca1 netdev_rxq_recv (ovs-vswitchd)
>                 #3  0x00000000005990b4 dp_netdev_process_rxq_port 
> (ovs-
> vswitchd)
>                 #4  0x000000000059981c pmd_thread_main (ovs-vswitchd)
>                 #5  0x000000000060ba34 ovsthread_wrapper 
> (ovs-vswitchd)
>                 #6  0x00007fcd9d47d52a start_thread (libpthread.so.0)
>                 #7  0x00007fcd9ca6b22d __clone (libc.so.6)
> 
>                 Stack trace of thread 6960:
>                 #0  0x00007fcd9ca5fc8d poll (libc.so.6)
>                 #1  0x0000000000637b9d time_poll (ovs-vswitchd)
>                 #2  0x000000000061e33c poll_block (ovs-vswitchd)
>                 #3  0x0000000000568c1f udpif_upcall_handler 
> (ovs-vswitchd)
>                 #4  0x000000000060ba34 ovsthread_wrapper 
> (ovs-vswitchd)
>                 #5  0x00007fcd9d47d52a start_thread (libpthread.so.0)
>                 #6  0x00007fcd9ca6b22d __clone (libc.so.6)
> 
> Our flavor like that:
> 
> 
> Any try can we do? Thank you.
> 
> 
> Best Regards
> Mao Ruoyu
> +86 21 38434078