[ovs-discuss] Assertions/crashes in lib/cmap.* on conntrack branch

Joe Stringer joe at ovn.org
Thu Dec 17 00:19:47 UTC 2015


Hi Ben,

Thanks for following up on this. Yes I think that all of the patches
we previously referred to are now merged in some form.

Is the core dump/backtrace the same as before? Could you repost the
backtrace in this thread?

Can you remind me what the port types are? Are bridges also being added/removed?

Jarno and I briefly spoke about this today, and one thought that came
up is whether the number of threads makes a difference here. Are you
also able to reproduce if you, for example, reduce the number of
revalidator/handler threads to 1?

ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1
ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1


On 16 December 2015 at 11:15, Ben Warren <ben at skyportsystems.com> wrote:
> Hi,
>
> We’re seeing this crash about a couple of times a day on our test bed,
> always when removing ports from a bridge (As Keith originally reported).  Do
> you have any idea what might be happening?  As I mentioned in my previous
> message, we’re pretty well at top of tree, so can very easily test any
> fixes.
>
> thanks,
> Ben
>
>
> On Dec 8, 2015, at 3:00 PM, Ben Warren <ben at skyportsystems.com> wrote:
>
> Hi Joe,
>
> Sorry for taking so long to get back to this.
>
> On Nov 23, 2015, at 6:54 PM, Joe Stringer <joe at ovn.org> wrote:
>
> On 20 November 2015 at 10:05, Keith Holleman <keith.holleman at gmail.com>
> wrote:
>
>
> Follow-up email here has the backtrace for the second method of
> reproduction.  In this case the bridge is not deleted, it was using the loop
> logic of effectively these commands:
>
>
> <snip>
>
> Thanks a lot for the report!
>
> Would you be able to apply these two patches and see if they fix the
> issue you are observing?
>
> https://patchwork.ozlabs.org/patch/541190/
> https://patchwork.ozlabs.org/patch/541191/
>
>
> Now that your conntrack code has been committed, we decided to build off the
> “openvswitch/ovs” repo on Github.  I built the top of the “branch-2.5”
> branch as of this morning:
>
> commit:
> https://github.com/openvswitch/ovs/commit/2862aeff82a3216ea4592c57299569484cf159ea
>
> and still see the crash.  The patches listed above do not apply cleanly: it
> looks like much (although maybe not all?) of the logic is already committed.
>
> Here’s what I see in /var/log/ovsswitchd.log:
>
> 2015-12-08T22:19:38.770Z|01159|bridge|INFO|bridge lan0: using datapath ID
> 0000dc39790002b0
> 2015-12-08T22:19:38.770Z|01160|connmgr|INFO|lan0: added service controller
> "punix:/var/run/openvswitch/lan0.mgmt"
> 2015-12-08T22:19:38.842Z|01161|dpif|WARN|system at ovs-system: failed to
> flow_del (No such file or directory)
> ufid:7580e732-908d-4134-9ca9-f6887195c2ae
> recirc_id(0),dp_hash(0),skb_priority(0),in_port(2),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=04:00:00:00:00:02,dst=04:00:00:00:00:fe),eth_type(0x0800),ipv4(src=192.168.27.2,dst=192.168.27.254,proto=6,tos=0,ttl=64,frag=no),tcp(src=39055,dst=11111),tcp_flags(psh|ack)
> 2015-12-08T22:19:38.842Z|01162|util|EMER|lib/cmap.c:846: assertion ok failed
> in cmap_replace()
> 2015-12-08T22:19:39.175Z|00002|daemon_unix(monitor)|ERR|1 crashes: pid 978
> died, killed (Aborted), core dumped, restarting
>
> System information:
>
> # ovs-ofctl --version
> ovs-ofctl (Open vSwitch) 2.5.0
> Compiled Dec  8 2015 12:16:49
> OpenFlow versions 0x1:0x4
>
> # uname -a
> Linux cd25 3.10.20-rt14-copilot #1 SMP Tue Dec 8 12:11:17 PST 2015 mips64
> GNU/Linux
>
> Please let us know what other information we can provide to help figure this
> out.
>
> regards,
> Ben
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> http://openvswitch.org/mailman/listinfo/discuss
>
>



More information about the discuss mailing list