[ovs-dev] [PATCH v7] ovsdb: provide raft and command interfaces with priority

[Ilya Maximets]

On 8/17/21 1:27 PM, Anton Ivanov wrote:
> Hi Ilia, hi list,
> 
> I ran some detailed experiments and there is an issue with all forms of "skipping" and/or reordering processing.
> 
> If the session list is skipped or reordered (I tried "fast-forwarding" the list to a new head position after hitting a time constraint), ovsdb fails to issue the response to some transactions when running the cluster test suite.
> 
> At present I am unable to get to the root cause.
> 
> The issue does not exist if processing bails out of the session loop and is re-run IN FULL (as in the earliest versions of the patch).

That is weird.  I'm not sure how the re-ordering is different from
the 're-run in full' here.  The only thing that different is an
actual order in which sessions are processed, because we're still
re-running all of them in full until the time allows.

> 
> I am going to re-issue the patch without any skipping whatsoever (either at remotes or at sessions level), because that works and improves raft (and overall ovn) stability.
> 
> While there may be some starvation of the sessions towards the end of the session list, it should be a second order effect, because re-processing sessions which have just been processed generates only a minimal amount of changes.
> 
> Skipping (if any) will be a later optimization after I get to the bottom of this and figure out why monitor updates are not followed by the transaction response.

This doesn't sound good to me.  It's pretty easy to spam the
ovsdb-server with monitor requests or condition changes.  This
requires walk across the whole database.  And if the database
is big enough, other sessions will never be served due to one
faulty/malicious connection.   It's also possible that we
have a few thousands connections and processing of all of them
legitimately takes a lot of time.  This will be a problem
if the rate of database changes is relatively high and constant.

Best regards, Ilya Maximets.