[ovs-dev] [PATCH] raft: Avoid sending equal snapshots.

[Han Zhou]

On Mon, May 25, 2020 at 6:11 AM Ilya Maximets <i.maximets at ovn.org> wrote:
>
> On 5/23/20 8:36 PM, Han Zhou wrote:
> >
> >
> > On Sat, May 23, 2020 at 10:34 AM Ilya Maximets <i.maximets at ovn.org
<mailto:i.maximets at ovn.org>> wrote:
> >>
> >> Snapshots are huge.  In some cases we could receive several outdated
> >> append replies from the remote server.  This could happen in high
> >> scale cases if the remote server is overloaded and not able to process
> >> all the raft requests in time.  As an action to each outdated append
> >> reply we're sending full database snapshot.  While remote server is
> >> already overloaded those snapshots will stuck in jsonrpc backlog for
> >> a long time making it grow up to few GB.  Since remote server wasn't
> >> able to timely process incoming messages it will likely not able to
> >> process snapshots leading to the same situation with low chances to
> >> recover.  Remote server will likely stuck in 'candidate' state, other
> >> servers will grow their memory consumption due to growing jsonrpc
> >> backlogs:
> >
> > Hi Ilya, this patch LGTM. Just not not clear about this last part of
the commit message. Why would remote server stuck in 'candidate' state if
there are pending messages from leader for it to handle? If the follower
was busy processing older messages, it wouldn't have had a chance to see
election timer timeout without receiving heartbeat from leader, so it
shouldn't try to start voting, right?
>
> I'm not sure what exactly happens, but that is what I see in my setup.
> Overloaded server sends vote requests almost each second with the term
> increased by 1 each time.  I think it doesn't see heartbeats since it
> processes only few messages at a time and a single message processing
> like applying the snapshot could lead to election timer expiration.
>
It processes at most 50 messages at a time for each connection in
raft_conn_run(), which should guarantee append_request (heartbeat) is seen.
However, it is possible that the connection is lost due to inactivity
probe, then append_request could be missed, causing re-election. Did you
see such case after applying the patch that disables inactivity probe for
raft connections?

>
> > Otherwise:
> >
> > Acked-by: Han Zhou <hzhou at ovn.org <mailto:hzhou at ovn.org>>
> >
> >>
> >> jsonrpc|INFO|excessive sending backlog, jsonrpc: ssl:192.16.0.3:6644 <
http://192.16.0.3:6644>,
> >>              num of msgs: 3795, backlog: 8838994624.
> >>
> >> This patch is trying to avoid that situation by avoiding sending of
> >> equal snapshot install requests.  This helps maintain reasonable memory
> >> consumption and allows the cluster to recover on a larger scale.
> >>
> >> Signed-off-by: Ilya Maximets <i.maximets at ovn.org <mailto:
i.maximets at ovn.org>>