[ovs-discuss] ovsdb-server core dump and ovsdb corruption using raft cluster

Yun Zhou yunz at nvidia.com
Fri Jul 13 22:49:49 UTC 2018


Hi,

We are running into some issues while we are trying out the 3 nodes raft ovsdb cluster in our lab, and hopefully we can get some help from the community. 

We are using ovs 2.9.2. 
-------------------------
 
We found that on one of the 3 nodes, the SB ovsdb-server was not started, and was not able to be restarted because its database was already corrupted:

   "ovsdb-server: syntax "{"encaps":["uuid","7f0f7605-c1d1-43fb-826a-1718ea70e088"],"hostname":"nd-sdn-dgx-010"}": syntax error: hostname is not a UUID"

Seeing from the ovsdb-server-sb log file history, SB ovsdb-server core dumped several days ago:

       "2018-07-08T06:58:15.267Z|00002|daemon_unix(monitor)|ERR|1 crashes: pid 937 died, killed (Aborted), core dumped, restarting"

Unfortunately, core dump was not generated.

FWIW, we saw core dumps for the NB ovsdb on all 3 cluster nodes, here is one of the stack:

(gdb) bt
#0  __GI_raise (sig=sig at entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fc48f8c2801 in __GI_abort () at abort.c:79
#2  0x00007fc48ff2c33c in ?? () from /usr/lib/x86_64-linux-gnu/libopenvswitch-2.9.so.0
#3  0x00007fc48ff2c2f2 in ?? () from /usr/lib/x86_64-linux-gnu/libopenvswitch-2.9.so.0
#4  0x00007fc48ff2e63c in json_to_ds ()
   from /usr/lib/x86_64-linux-gnu/libopenvswitch-2.9.so.0
#5  0x00007fc4902ed50f in ovsdb_log_compose_record ()
   from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#6  0x00007fc4902ed7ef in ovsdb_log_write ()
   from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#7  0x00007fc4902ed96e in ovsdb_log_write_and_free ()
   from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#8  0x00007fc4902f3684 in ?? () from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#9  0x00007fc4902f3bf3 in ?? () from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#10 0x00007fc4902fb7bd in raft_store_snapshot ()
   from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#11 0x00007fc4903027ae in ?? () from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#12 0x00007fc4903031de in ovsdb_storage_store_snapshot ()
   from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#13 0x00007fc4902efcab in ovsdb_snapshot ()
   from /usr/lib/x86_64-linux-gnu/libovsdb-2.9.so.0
#14 0x0000561e47a8cf82 in ?? ()
#15 0x00007fc48f8a3b97 in __libc_start_main (main=0x561e47a8bef0, argc=17, 
    argv=0x7ffe000ce2c8, init=<optimized out>, fini=<optimized out>, 
    rtld_fini=<optimized out>, stack_end=0x7ffe000ce2b8) at ../csu/libc-start.c:310
#16 0x0000561e47a8db9a in ?? ()

Please let us know if any more information is needed. Thanks very much!

- Yun


-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------


More information about the discuss mailing list