[ovs-dev] [PATCH 0/5] Stream Record/Replay.

[Dumitru Ceara]

On 7/14/20 4:56 PM, Ilya Maximets wrote:
> On 7/14/20 3:34 PM, Dumitru Ceara wrote:
>> On 6/30/20 3:24 AM, Ilya Maximets wrote:
>>> This patch set adds new stream provider and other functionality in
>>> order to record all the incoming data on all the steams (ssl, tcp,
>>> unixctl) of openvswitch library based applications and replay these
>>> streams later for debugging purposes or performance tests.
>>>
>>> For example, these changes allowed me to record the lifecycle of
>>> a standalone ovsdb-server in a fake-multinode cluster from ovn
>>> scale testing setup with 100 nodes.  While having only replay files,
>>> user could reproduce transactions and connections in a big cluster on
>>> a local PC wile being able to tweak various log levels, run everything
>>> under debugger or tracer, measure performance difference with perf.
>>>
>>> Current implementation doesn't work with clustered databases since
>>> raft heavily depends on time events.  This is a point of further
>>> improvement.
>>>
>>> Some more details in individual commit messages.  Some documentation
>>> updates are missing, will be in next version of the patch-set.  Code
>>> maybe not that elegant in a couple of places, I'm going to work on
>>> this too.
>>>
>>> ovn-northd might be a good candidate to integrate this functionality
>>> from the OVN side.
>>>
>>> Ilya Maximets (5):
>>>   ovsdb-server: Allow using predictable UUIDs.
>>>   stream: Add record/replay functionality.
>>>   ovsdb-server: Integrate stream replay engine.
>>>   ovsdb-server: Don't update manager status if replay engine is active.
>>>   jsonrpc: Disable inactivity probes if replay engine is active.
>>>
>>>  lib/automake.mk         |   1 +
>>>  lib/jsonrpc.c           |  10 +-
>>>  lib/stream-provider.h   |   4 +
>>>  lib/stream-replay.c     | 561 ++++++++++++++++++++++++++++++++++++++++
>>>  lib/stream.c            |  38 ++-
>>>  lib/stream.h            |  46 +++-
>>>  lib/uuid.c              |  34 +++
>>>  lib/uuid.h              |   1 +
>>>  ovsdb/ovsdb-client.c    |   2 +-
>>>  ovsdb/ovsdb-server.c    |  30 ++-
>>>  tests/test-jsonrpc.c    |   2 +-
>>>  utilities/ovs-vsctl.c   |   2 +-
>>>  vswitchd/ovs-vswitchd.c |   2 +-
>>>  vtep/vtep-ctl.c         |   2 +-
>>>  14 files changed, 717 insertions(+), 18 deletions(-)
>>>  create mode 100644 lib/stream-replay.c
>>>
>>
>> Hi Ilya,
>>
>> This is really great and it will make performance testing easier indeed!
>>
>> This is not a complete review as I just played a bit with the
>> functionality and recorded and replayed some OVN DB executions but
>> something that I missed was a bit of documentation or maybe a short
>> example (apart from the samples in the commit logs).
> 
> Yeah.  I know, this needs some chapter in documentation, otherwise it's
> not obvious how to use this functionality in a correct way.  I'll work
> on this.
> 
>>
>> Also, I'm wondering if there's any way possibility to expand this
>> feature and allow users to start/stop recording while the application is
>> already running. If that could be done in a reliable way it would make
>> it usable "online", potentially even in production.
> 
> This is definitely a great feature to have.
> There are few points here that we should take care of in case we would
> like to have recording being enabled in runtime:
> 1. Randomness: we need to be able to re-initialize UUID generator
>    in run-time, i.e. force it to start generation of non-random values.

Isn't it an option to record generated UUIDs too? And, instead of
regenerating them, just replay them in "replay" mode.

> 2. Need to pause all the connections and record their initial states.
> 3. Need to dump current DB state to be able to start replay on it.
> 4. We'll probably need to resolve issues with time-based events first,
>    or being able to disable them in runtime.

Ack.

> 
> Maybe it's not a full list, but this already will require some amount
> of work.  However, this might be possible.
> 

That would be great!

Regards,
Dumitru

> One implementation idea is that we could have callbacks registered for
> each part of the code that interested in record/replay and these
> callbacks could be called to record initial states while enabling
> recording in runtime. E.g. one callback for UUID generator, one for
> each running stream, one for DB storage, etc.
> 
> However, there are things that depends on a sequences of events that
> could be hard to restore without having exact same messages being
> processed in exact same order.  For example, state of the ovsdb-idl
> state machine or something like that.  We will probably end up
> storing big chunk of a process memory to keep things running as
> expected on replay.
> 
>>
>> I have a few minor comments on the individual patches.
>>
>> Again, great work!
> 
> Thanks for checking it out!
> 
>>
>> Thanks,
>> Dumitru
>>
>