[ovs-dev] [ovsdb sppedup v2 00/19] improve ovsdb connection scaling

[Andy Zhou]

[ This patch set is mirrored at: branch 'ovsdb-speedup' of 'github.com/azhou-nicira/ovs-review'. ]

This patch set implements two ideas improve ovsdb connection scaling:

* Allow multiple jsonrpc connection to share a single ovsdb monitor, thus
  maintaining a single set of changes.

* With in the same monitor, cache the json object generated for an update.
  If the same update is requested by another jsonrpc connection, the
  cached json object can be used, avoid regenerating it from scratch.


Scaling improvments
===================
Focusing on measuring the delay of multiple back to back transactions, I used
'ovs-vsct add-port br0 p1 -- add-port br0 p2 -- ...   add-port br0 p20'
to add 20 ports in one go, and measure the delay in executing the
entire command. This would show ovsdb-server's work load of not only
handling each transaction, but also the background work required to
finish servicing the transaction.

       Here are the results:

       Without this patch series:
       ==================
       #monitors       delay of 20 transactions
       1                       0.046s
       100                   0.532s
       200                   1.007s
       300                   1.599s

       With this patch series:
       =================
       #monitors       delay of 20 transactions
       1                       0.046s
       100                   0.047s
       200                   0.047s
       300                   0.049s


Version history:
===============
v1->v2:   Added multiple clients to backlogged test case.
          maintaining 'changes' in hmap instead of linked-list.
          Each monitor owns its own json cache. Remove the glbal json cache.
          Valgrind tested all ovsdb related test (911 - 1630)
          style and typo fixes


Andy Zhou (19):
  ovsdb-test: add multiple clients to backlogged connection test
  jsonrpc-server: split monitors into database back end and JSON-RPM
    front     end
  jsonrpc-server: make setting mt->select into its own functions
  jsonrpc-server: refactor ovsdb_jsonrpc_parse_monitor_request
  jsonrpc-server: refactor ovsdb_monitor_add_column()
  jsonrpc-server: refactoring
    ovsdb_jsonrpc_monitor_compose_table_update()
  jsonrpc-server: refactoring ovsdb_jsonrpc_monitor_needs_flush
  jsonrpc-server: rename ovsdb_jsonrpc_monitor_get_initial()
  jsonrpc-server: refactoring ovsdb_monitor_destroy()
  jsonrpc-server: Split out monitor backend functions to monitor.c/h
  ovsdb-monitor: refactoring ovsdb_monitor_get_initial
  ovsdb: ovsdb-monitor stores jsonrpc-monitor in a linked-list
  ovsdb-monitor: add transaction ids
  ovsdb-monitor: rename jsonrpc_monitor_compose_table_update()
  ovsdb-monitor: add ovsdb_monitor_changes
  ovsdb-monitor: allow multiple of 'ovsdb_monitor_changes' in each ovsdb
    monitor table
  ovsdb-monitor: refactor ovsdb_monitor_create()
  ovsdb-monitor: allow multiple jsonrpc monitors to share a single ovsdb
    monitor
  ovsdb-monitor: add json cache

 ovsdb/automake.mk      |   2 +
 ovsdb/jsonrpc-server.c | 570 ++++------------------------
 ovsdb/jsonrpc-server.h |   3 +
 ovsdb/monitor.c        | 985 +++++++++++++++++++++++++++++++++++++++++++++++++
 ovsdb/monitor.h        |  68 ++++
 tests/ovsdb-server.at  |  16 +-
 6 files changed, 1140 insertions(+), 504 deletions(-)
 create mode 100644 ovsdb/monitor.c
 create mode 100644 ovsdb/monitor.h

-- 
1.9.1