[ovs-discuss] [ovs-dev] issue with ovs_rcu corrupted cbset (fwd)

[Sabyasachi Sengupta]

---------- Forwarded message ----------
Date: Wed, 29 Jul 2015 17:02:13 -0700
From: Sabyasachi Sengupta <Sabyasachi.Sengupta at alcatel-lucent.com>
To: ovs-dev <dev at openvswitch.org>
Subject: [ovs-dev] issue with ovs_rcu corrupted cbset


We recently came across a case where ovs trips due to a bad function pointer 
because of a possible corruption in ovs rcu cbset. This typically happens if 
we have 100+ VMs with ~1000 flows.

(gdb) p *cbset
$16 = {list_node = {prev = 0x7f69c0e77b70, next = 0x208f730}, cbs = {{
        function = 0x416e80 <rule_destroy_cb>, aux = 0x461d540}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x45e1d40}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x29cb240}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x1c05c60}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x215c410}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x20cdca0}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x2224ce0}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x29c8710}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x29c8a80}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x2492670}, {
        function = 0x198e1b8, aux = 0x25832f0}, { <<<<<
        function = 0x416e80 <rule_destroy_cb>, aux = 0x25adc80}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x39cb7d0}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x39cb1f0}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x39cb4e0}, {
        function = 0x416e80 <rule_destroy_cb>, aux = 0x292b130}}, n_cbs = 16}

We see that vswitchd trips with the below stacktrace.

Thread 1 (Thread 0x7f69c0e78700 (LWP 5308)):
#0  0x000000000198e1b8 in ?? ()
#1  0x0000000000499430 in ovsrcu_call_postponed () at lib/ovs-rcu.c:249
#2  0x0000000000499674 in ovsrcu_postpone_thread (arg=<value optimized out>)
      at lib/ovs-rcu.c:265
#3  0x00000000004998f1 in ovsthread_wrapper (aux_=<value optimized out>)
      at lib/ovs-thread.c:322
#4  0x00000030b80079d1 in start_thread () from /lib64/libpthread.so.0
#5  0x00000030b78e8b5d in clone () from /lib64/libc.so.6

We are on ovs-2.3.1. Is this problem fixed in more recent ovs (upcoming 2.4) 
or in master? If so, can anyone point me to the patch or commit hash that we 
can try?
_______________________________________________
dev mailing list
dev at openvswitch.org
http://openvswitch.org/mailman/listinfo/dev