[ovs-dev] [PATCH] ovsdb raft: Sync commit index to followers without delay.

Han Zhou zhouhan at gmail.com
Fri Mar 15 22:37:20 UTC 2019


On Fri, Mar 15, 2019 at 3:30 PM Ben Pfaff <blp at ovn.org> wrote:
>
> On Thu, Mar 14, 2019 at 10:13:56AM -0700, Han Zhou wrote:
> > From: Han Zhou <hzhou8 at ebay.com>
> >
> > When update is requested from follower, the leader sends AppendRequest
> > to all followers and wait until AppendReply received from majority, and
> > then it will update commit index - the new entry is regarded as committed
> > in raft log. However, this commit will not be notified to followers
> > (including the one initiated the request) until next heartbeat (ping
> > timeout), if no other pending requests. This results in long latency
> > for updates made through followers, especially when a batch of updates
> > are requested through the same follower.
> >
> > $ time for i in `seq 1 100`; do ovn-nbctl ls-add ls$i; done
> >
> > real    0m34.154s
> > user    0m0.083s
> > sys 0m0.250s
> >
> > This patch solves the problem by sending heartbeat as soon as the commit
> > index is updated in leader. It also avoids unnessary heartbeat by resetting
> > the ping timer whenever AppendRequest is broadcasted. With this patch
> > the performance is improved more than 50 times in same test:
> >
> > $ time for i in `seq 1 100`; do ovn-nbctl ls-add ls$i; done
> >
> > real    0m0.564s
> > user    0m0.080s
> > sys 0m0.199s
> >
> > The parameters in torture test is also adjusted because of the improved
> > performance, otherwise the tests will all be skipped.
>
> The patch seems very reasonable and the concept makes sense, but when I
> run
>         make -j10 check TESTSUITEFLAGS='-k ovsdb,cluster,torture -j10'
> it comes near to killing my laptop, with multiple ovsdb-servers going to
> 100% CPU.  Without the patch, I don't see behavior like that at all.
>
> Do you see the same thing?

I think this is caused by the change in torture test case, as
mentioned in the commit message: The parameters in torture test is
also adjusted because of the improved
performance, otherwise the tests will all be skipped. (because all
clients finishes the tasks at phase 0)

Unlike other tests, I never run torture tests using -j, since it is
more related to timing and less stable. Now since I increased the
clients to 20 x 40 in the test, it is likely to kill your laptop by
using -j10. Do you have any suggestion for this? Maybe I can try keep
the number of clients small but put some sleep between requests to
slow them down.


More information about the dev mailing list