[ovs-dev] [PATCH v3 0/9] Stream Record/Replay.

[Ilya Maximets]

This patch set adds new stream provider and other functionality in
order to record all the incoming data on all the steams (ssl, tcp,
unixctl) of openvswitch library based applications and replay these
streams later for debugging purposes or performance tests.

For example, these changes allowed me to record the lifecycle of
a standalone ovsdb-server in a fake-multinode cluster from ovn
scale testing setup with 100 nodes.  While having only replay files,
user could reproduce transactions and connections in a big cluster on
a local PC wile being able to tweak various log levels, run everything
under debugger or tracer, measure performance difference with perf.

It also proven to be useful for debugging issues in ovn components.
IIRC, following issue was not easily reproducible, but it was recorded
and debugged by using v1 of this patch series integrated into
ovn-controller:

  91a6a4580267 ("ovsdb-idl: Fix use-after-free when deleting orphaned rows.")

Current implementation doesn't work with clustered databases since
raft heavily depends on time events.  This is a point of further
improvement.

More details and usage examples in the documentation in the last
patch of this set.

Version 3:
  - Addressed comments from Dumitru:
    - Changed '--replay-record' option to just '--record'.
    - Fixed 'name' leak in new_replay_stream(), removed incorrect comments.
    - Bunch of small style fixes.
    - Rate-limited all replay errors.
  - Added Acked-by from Dumitru to all patches except for patch #2.

Version 2:
  - Fixed recording of listen/accept/open failures.
  - Fixed recording of client connections.
  - Replay file handling split out of stream code.
  - Replay files could be stored in a separate directory now.
  - Implemented recording of generated UUIDs. Removed support for
    generation of predictable UUIDs since not required anymore.
  - Added man pages and additional user documentation.
  - Record/replay integrated into ovsdb-client.
  - Added unit tests for both server and client cases.
  - Various code improvements.
  - I did not add storing of cmdline arguments and copying of
    a database file, because it's very easy to implement inside
    of ovn-ctl, but not that easy to implement in ovsdb-server.

Version 1:
  - https://patchwork.ozlabs.org/project/openvswitch/list/?series=186549&state=*

Ilya Maximets (9):
  ovs-replay: New library to create and manage replay files.
  stream: Add record/replay functionality.
  uuid: Allow record/replay of generated UUIDs.
  ovsdb-server: Integrate stream replay engine.
  ovsdb-server: Don't update manager status if replay engine is active.
  jsonrpc: Disable inactivity probes if replay engine is active.
  ovsdb-server.at: Add unit test for record/replay.
  ovsdb-client: Integrate record/replay functionality.
  docs: Add a topic about record/replay with ovsdb-server.

 Documentation/automake.mk              |   1 +
 Documentation/topics/index.rst         |   1 +
 Documentation/topics/record-replay.rst | 138 ++++++++
 NEWS                                   |   4 +
 lib/automake.mk                        |   6 +
 lib/jsonrpc.c                          |  11 +-
 lib/ovs-replay-syn.man                 |   3 +
 lib/ovs-replay.c                       | 243 +++++++++++++
 lib/ovs-replay.h                       | 163 +++++++++
 lib/ovs-replay.man                     |  16 +
 lib/ovs-replay.xml                     |  35 ++
 lib/stream-provider.h                  |   5 +
 lib/stream-replay.c                    | 456 +++++++++++++++++++++++++
 lib/stream.c                           |  35 +-
 lib/stream.h                           |  12 +
 lib/uuid.c                             |  76 +++++
 ovsdb/ovsdb-client.1.in                |   2 +
 ovsdb/ovsdb-client.c                   |   5 +
 ovsdb/ovsdb-server.1.in                |   2 +
 ovsdb/ovsdb-server.c                   |  16 +-
 tests/ovsdb-client.at                  |  89 +++++
 tests/ovsdb-server.at                  | 149 ++++++++
 22 files changed, 1458 insertions(+), 10 deletions(-)
 create mode 100644 Documentation/topics/record-replay.rst
 create mode 100644 lib/ovs-replay-syn.man
 create mode 100644 lib/ovs-replay.c
 create mode 100644 lib/ovs-replay.h
 create mode 100644 lib/ovs-replay.man
 create mode 100644 lib/ovs-replay.xml
 create mode 100644 lib/stream-replay.c

-- 
2.26.3