[ovs-dev] Coallescing of monitor notifications

[Dennis Sam]

On Sat, Nov 5, 2016 at 9:59 AM, Ben Pfaff <blp at ovn.org> wrote:

> On Thu, Nov 03, 2016 at 05:28:15PM -0700, Dennis Sam wrote:
> > After upgrading to 2.5.x, we noticed that monitor notifications can now
> > coalesce multiple transactions. Can this be confirmed as new behaviour
> and
> > that transactions are indeed being coalesced and not simply combined?
>
> We did add that feature in some version, in commit 48f6e410db6b2
> (jsonrpc-server: Combine notifications when connection becomes
> backlogged.).  The commit message was:
>
>     jsonrpc-server: Combine notifications when connection becomes
> backlogged.
>
>     Connections that queue up too much data, because they are monitoring a
>     table that is changing quickly and failing to keep up with the updates,
>     cause problems with buffer management.  Since commit 60533a405b2e
>     (jsonrpc-server: Disconnect connections that queue too much data.),
>     ovsdb-server has dealt with them by disconnecting the connection and
>     letting them start up again with a fresh copy of the database.
> However,
>     this is not ideal because of situations where disconnection happens
>     repeatedly.  For example:
>
>          - A manager toggles a column back and forth between two or more
> values
>            quickly (in which case the data transmitted over the monitoring
>            connections always increases quickly, without bound).
>
>          - A manager repeatedly extends the contents of some column in
> some row
>            (in which case the data transmitted over the monitoring
> connection
>            grows with O(n**2) in the length of the string).
>
>     A better way to deal with this problem is to combine updates when they
> are
>     sent to the monitoring connection, if that connection is not keeping
> up.
>     In both the above cases, this reduces the data that must be sent to a
>     manageable amount.  This commit implements this new way.
>
>     Bug #1211786.
>     Bug #1221378.
>     Signed-off-by: Ben Pfaff <blp at nicira.com>
>     Acked-by: Andy Zhou <azhou at nicira.com>
>
> I don't think that your distinction between coalescing and combining is
> a standard one.
>
I was try to make a distinction between a monitor notification that simply
colates the notifications into a large one versus sending the new update
based on the delta between database states between notifications. Since the
monitor notification is key'ed by UUID I suppose it must coalese/combine.


>
> > Example 1:
> >
> > T0 Add A with uuid 0
> > T1 Del A with uuid 0
> > T2 Add B with uuid 1
> >
> > Monitor notification:
> > Add B with uuid 1
>
> Yes, that's what ovsdb-server would send, if it had to coalesce/combine
> the transactions.
>
> > Example 2:
> >
> > T0 Add A with uuid 0
> > T1 Upd A with uuid 0 to A'
> >
> >
> > Monitor notification:
> > Adds A' with uuid 0
>
> Ditto.
>
> > There are probably other examples but this was to illustrate my point
> about
> > coalescing the transactions.
> >
> > Is there a knob to enable/disable this behaviour?
>
> No.
>
> Is there something about this behavior that could disturb a client?  It
> should generally be transparent, especially if one considers that it's
> equivalent to what the client would see if it lost a connection to the
> server and then reconnected.
>
The notion of coalescing updates is a fine and a validate way to handle
flappy state change as you described, I don't think it breaks the general
notification monitor mechanism. We noticed this when managing a table that
is keyed by it's OVSDB enforced key (as opposed to UUID) before processing
the entire monitor notification.

For example, but given

T0 Add A with key X and uuid 0
Map:
X <> A
UUID 0 <> A

T1 Del A with key X and uuid 0
T2 Add B with key X and uuid 1

Assuming T1 and T2 are combined, and you process T2 before T1 (traversal of
UUID keys in monitor notification doesn't enforce Tx order), you can get
UUID 1 <> B but loose the key to object mapping when T1 is processed.

I suppose a client can always create a transaction with multiple operations
in a similar way so coalescing updates shouldn't be introducing anything
new.

This is just something not obvious and expected.

Thanks for confirming Ben.