[ovs-discuss] Possible data loss of OVSDB active-backup mode

Numan Siddique nusiddiq at redhat.com
Thu Aug 9 08:35:06 UTC 2018


On Thu, Aug 9, 2018 at 1:07 AM Ben Pfaff <blp at ovn.org> wrote:

> On Wed, Aug 08, 2018 at 12:18:10PM -0700, Han Zhou wrote:
> > On Wed, Aug 8, 2018 at 11:24 AM, Ben Pfaff <blp at ovn.org> wrote:
> > >
> > > On Wed, Aug 08, 2018 at 12:37:04AM -0700, Han Zhou wrote:
> > > > Hi,
> > > >
> > > > We found an issue in our testing (thanks aginwala) with active-backup
> > mode
> > > > in OVN setup.
> > > > In the 3 node setup with pacemaker, after stopping pacemaker on all
> > three
> > > > nodes (simulate a complete shutdown), and then if starting all of
> them
> > > > simultaneously, there is a good chance that the whole DB content gets
> > lost.
> > > >
> > > > After studying the replication code, it seems there is a phase that
> the
> > > > backup node deletes all its data and wait for data to be synced from
> the
> > > > active node:
> > > >
> https://github.com/openvswitch/ovs/blob/master/ovsdb/replication.c#L306
> > > >
> > > > At this state, if the node was set to active, then all data is gone
> for
> > the
> > > > whole cluster. This can happen in different situations. In the test
> > > > scenario mentioned above it is very likely to happen, since pacemaker
> > just
> > > > randomly select one as master, not knowing the internal sync state of
> > each
> > > > node. It could also happen when failover happens right after a new
> > backup
> > > > is started, although less likely in real environment, so starting up
> > node
> > > > one by one may largely reduce the probability.
> > > >
> > > > Does this analysis make sense? We will do more tests to verify the
> > > > conclusion, but would like to share with community for discussions
> and
> > > > suggestions. Once this happens it is very critical - even more
> serious
> > than
> > > > just no HA. Without HA it is just control plane outage, but this
> would
> > be
> > > > data plane outage because OVS flows will be removed accordingly since
> > the
> > > > data is considered as deleted from ovn-controller point of view.
> > > >
> > > > We understand that active-standby is not the ideal HA mechanism and
> > > > clustering is the future, and we are also testing the clustering with
> > the
> > > > latest patch. But it would be good if this problem can be addressed
> with
> > > > some quick fix, such as keep a copy of the old data somewhere until
> the
> > > > first sync finishes?
> > >
> > > This does seem like a plausible bug, and at first glance I believe that
> > > you're correct about the race here.  I guess that the correct behavior
> > > must be to keep the original data until a new copy of the data has been
> > > received, and only then atomically replace the original by the new.
> > >
> > > Is this something you have time and ability to fix?
> >
> > Thanks Ben for quick response. I guess I will not have time until I send
> > out next series for incremental processing :)
> > It would be good if someone can help and then please reply this email if
> > he/she starts working on it so that we will not end up with overlapping
> > work.
>

I will give a shot at fixing this issue.

In the case of tripleo we haven't hit this issue. I haven't tested this
scenario.
I will test it out. One difference when compared to your setup is tripleo
uses
IPAddr2 resource and a collocation constraint set.

Thanks
Numan



> >
> > One more thing that confuses me in the code is:
> >
> https://github.com/openvswitch/ovs/blob/master/ovsdb/ovsdb-server.c#L213-L216
> > Does this code just change the ovsdb-server instance from backup to
> active
> > when the connection to active is lost? This isn't the behavior described
> by
> > manpage of active-standby mode. Isn't the mode supposed to be changed
> only
> > by management softeware/human?
>
> replication_is_alive() is based on jsonrpc_session_is_alive(), which
> returns true if the session is not permanently dead, that is, if it's
> currently connected or trying to reconnect.  It really only returns
> false if a maximum number of retries was configured (by default, there
> is none) and that number has been exceeded.
>
> I'm actually not sure whether there's a way that this particular
> jsonrpc_session_is_alive() could ever return false.  Maybe only if the
> session was started with a remote that fails parsing?  Maybe it should
> not be checked at all.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20180809/35b3c629/attachment.html>


More information about the discuss mailing list