[ovs-dev] [PATCH v2 0/2] Mitigate RAFT memory consumption issues.

[Ilya Maximets]

On 11/3/20 4:53 PM, Ilya Maximets wrote:
> Version 2:
>  - Skipping 3 patches that already applied.
>  - Fixed use of uninitialized backlog limits in jsonrpc_session,
>    i.e. added initialization on each jsorpc_session_open*()
>  - Added Acked-by from Dumitru to the last patch.
> 
> Original cover letter:
> 
> Under a heavy load or in a long run memory consumption if ovsdb-server
> process that is part of a RAFT cluster could reach very high values.
> From my experience it could be up to 60-100 GB.  In these conditions
> it's likely that ovsdb-server will be killed by OOM-killer or just
> will not be able to work properly wasting time on processing outdated
> or unneeded data.  There are 3 main parts that consumes most of the
> memory:
> 
>  1. Backlog on RAFT connections between servers.
>  2. Local RAFT log.
>  3. Libc doesn't return memory back to system.
> 
> Backlog could start growing if one of remote servers doesn't doing
> well and is not able to process requests in time.  This sending
> backlog could contain snapshots or even just big number of big
> append requests.  It could grow to tens of GBs really fast and
> most of this data might be even unnecessary if it becomes obsolete
> by one of the previous requests or if current 'term' changes and
> all the old messages should be dropped.  Solution for this is
> to monitor the size of the current backlog and disconnect if it grows
> too big since it will be easier to just reconnect and send one new
> snapshot.
> 
> Local RAFT log contains all the DB changes that are not part of a
> snapshot yet.  Since snapshots are taken at most once in 10 minutes,
> log could grow pretty big.  Up to tens of thousands of entries and
> each of these entries could be fairly big by themselves.  That being
> said RAFT log could grow up to tens of GBs too.
> 
> One extra point for memory consumption is that memory likely doesn't
> go away even after calling free() due to implementation of a C memory
> allocators.  And this happens a lot.  ovsdb-server process usually
> holds a lot of system memory even if the database is almost empty.
> This heap memory might be returned back to OS by using malloc_trim().
> 
> --
> All of these issues was found on branch-2.13, but it always hard to
> distinguish new features from the bug fix when we're talking about
> scaling issues.  Anyway, I think, it'll be good to have these
> patches (if they are any good) backorted to 2.13, especially because
> it's going to be our next LTS.  Thoughts?
> 
> 
> Ilya Maximets (2):
>   raft: Set threshold on backlog for raft connections.
>   raft: Make backlog thresholds configurable.
> 
>  NEWS                    |  3 +++
>  lib/jsonrpc.c           | 60 ++++++++++++++++++++++++++++++++++++++++-
>  lib/jsonrpc.h           |  6 +++++
>  ovsdb/ovsdb-server.1.in |  5 ++++
>  ovsdb/raft.c            | 50 ++++++++++++++++++++++++++++++++++
>  5 files changed, 123 insertions(+), 1 deletion(-)
> 

Dumitru, thanks for review!

Applied to master and backported down to 2.13.

Best regards, Ilya Maximets.