[ovs-dev] [PATCH 0/5] Mitigate RAFT memory consumption issues.

[Ilya Maximets]

Under a heavy load or in a long run memory consumption if ovsdb-server
process that is part of a RAFT cluster could reach very high values.
>From my experience it could be up to 60-100 GB.  In these conditions
it's likely that ovsdb-server will be killed by OOM-killer or just
will not be able to work properly wasting time on processing outdated
or unneeded data.  There are 3 main parts that consumes most of the
memory:

 1. Backlog on RAFT connections between servers.
 2. Local RAFT log.
 3. Libc doesn't return memory back to system.

Backlog could start growing if one of remote servers doesn't doing
well and is not able to process requests in time.  This sending
backlog could contain snapshots or even just big number of big
append requests.  It could grow to tens of GBs really fast and
most of this data might be even unnecessary if it becomes obsolete
by one of the previous requests or if current 'term' changes and
all the old messages should be dropped.  Solution for this is
to monitor the size of the current backlog and disconnect if it grows
too big since it will be easier to just reconnect and send one new
snapshot.

Local RAFT log contains all the DB changes that are not part of a
snapshot yet.  Since snapshots are taken at most once in 10 minutes,
log could grow pretty big.  Up to tens of thousands of entries and
each of these entries could be fairly big by themselves.  That being
said RAFT log could grow up to tens of GBs too.

One extra point for memory consumption is that memory likely doesn't
go away even after calling free() due to implementation of a C memory
allocators.  And this happens a lot.  ovsdb-server process usually
holds a lot of system memory even if the database is almost empty.
This heap memory might be returned back to OS by using malloc_trim().

--
All of these issues was found on branch-2.13, but it always hard to
distinguish new features from the bug fix when we're talking about
scaling issues.  Anyway, I think, it'll be good to have these
patches (if they are any good) backorted to 2.13, especially because
it's going to be our next LTS.  Thoughts?

Ilya Maximets (5):
  raft: Add log length to the memory report.
  ovsdb-server: Reclaim heap memory after compaction.
  raft: Set threshold on backlog for raft connections.
  raft: Make backlog thresholds configurable.
  raft: Avoid having more than one snapshot in-flight.

 NEWS                    |  6 +++
 configure.ac            |  1 +
 lib/jsonrpc.c           | 57 ++++++++++++++++++++++++-
 lib/jsonrpc.h           |  6 +++
 ovsdb/ovsdb-server.1.in |  9 ++++
 ovsdb/ovsdb-server.c    | 41 +++++++++++++++++-
 ovsdb/ovsdb.c           | 12 +++++-
 ovsdb/ovsdb.h           |  3 +-
 ovsdb/raft-private.c    |  1 -
 ovsdb/raft-private.h    |  4 +-
 ovsdb/raft.c            | 93 +++++++++++++++++++++++++++++------------
 11 files changed, 199 insertions(+), 34 deletions(-)

-- 
2.25.4