[ovs-discuss] [ovs-dev] intermittent ovs-vswitchd segfault

[Numan Siddique]

On Wed, Jan 11, 2017 at 5:24 AM, Joe Stringer <joe at ovn.org> wrote:

> On 10 January 2017 at 00:25, Numan Siddique <nusiddiq at redhat.com> wrote:
> > Hi,
> >
> > I am seeing intermittent segfault's in ovs-vswitchd. We have like 20
> > compute nodes and noticed the crash in 4 or 5 nodes. Seems to me the
> crash
> > is seen when the system is idle for a long time (as I noticed over the
> > weekend)
> >
> > We are using master of ovs (with the latest commit id 92043ab8ffd4)
> >
> > Below is the dmesg and the backtrace of the core file
> >
> > --------
> > [297752.801094] revalidator239[2308]: segfault at 0 ip 00007f2a8d966da3
> sp
> > 00007f2a28ff6c58 error 4 in ovs-vswitchd[7f2a8d8ef000+1ea000]
> > ------------
> >
> >
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock
> > -vconsole:emer -vsyslog:err -vfi'.
> > Program terminated with signal 11, Segmentation fault.
> > #0  0x00007fe969b13da3 in cmap_replace__ ()
> > Missing separate debuginfos, use: debuginfo-install
> > glibc-2.17-157.el7_3.1.x86_64 keyutils-libs-1.5.8-3.el7.x86_64
> > krb5-libs-1.14.1-27.el7_3.x86_64 libcap-ng-0.7.5-4.el7.x86_64
> > libcom_err-1.42.9-9.el7.x86_64 libselinux-2.5-6.el7.x86_64
> > openssl-libs-1.0.1e-60.el7.x86_64 pcre-8.32-15.el7_2.1.x86_64
> > zlib-1.2.7-17.el7.x86_64
> > (gdb) br
> > Breakpoint 1 at 0x7fe969b13da3
> > (gdb) bt
> > #0  0x00007fe969b13da3 in cmap_replace__ ()
> > #1  0x00007fe969b14491 in cmap_replace ()
> > #2  0x00007fe969aee9ff in ukey_delete ()
> > #3  0x00007fe969aefd42 in revalidator_sweep__ ()
> > #4  0x00007fe969af1bad in udpif_revalidator ()
> > #5  0x00007fe969b8b2a6 in ovsthread_wrapper ()
> > #6  0x00007fe968e07dc5 in start_thread () from /lib64/libpthread.so.0
> > #7  0x00007fe96862c73d in clone () from /lib64/libc.so.6
> >
> > Thanks
> > Numan
> > _______________________________________________
> > dev mailing list
> > dev at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
>
> Hi Numan,
>
> Thanks for the report.
>
> First, a couple of questions:
> * Was there anything relevant in the ovs-vswitchd log?
>

​Couldn't find any. I will report it if i can find anything relevant.
​


> * Can you provide backtraces for the other threads?
>
> ​Please see this paste which has the bt of all threads -
https://paste.fedoraproject.org/525354/
​


> Jarno and I discussed this, and there's some possibility that this
> patch will fix the issue. If you're able to test, please let us know
> whether this makes a difference:
> https://mail.openvswitch.org/pipermail/ovs-dev/2017-January/327513.html
>


​Thanks for the patch. I will try to test it out and let you know.

Thanks
Numan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20170111/019f6b55/attachment.html>