[ovs-dev] [PATCH 0/5] Mitigate RAFT memory consumption issues.
Ilya Maximets
i.maximets at ovn.org
Tue Nov 3 15:55:43 UTC 2020
On 10/28/20 12:03 PM, Dumitru Ceara wrote:
> On 10/26/20 2:42 AM, Ilya Maximets wrote:
>> Under a heavy load or in a long run memory consumption if ovsdb-server
>> process that is part of a RAFT cluster could reach very high values.
>> From my experience it could be up to 60-100 GB. In these conditions
>> it's likely that ovsdb-server will be killed by OOM-killer or just
>> will not be able to work properly wasting time on processing outdated
>> or unneeded data. There are 3 main parts that consumes most of the
>> memory:
>>
>> 1. Backlog on RAFT connections between servers.
>> 2. Local RAFT log.
>> 3. Libc doesn't return memory back to system.
>>
>> Backlog could start growing if one of remote servers doesn't doing
>> well and is not able to process requests in time. This sending
>> backlog could contain snapshots or even just big number of big
>> append requests. It could grow to tens of GBs really fast and
>> most of this data might be even unnecessary if it becomes obsolete
>> by one of the previous requests or if current 'term' changes and
>> all the old messages should be dropped. Solution for this is
>> to monitor the size of the current backlog and disconnect if it grows
>> too big since it will be easier to just reconnect and send one new
>> snapshot.
>>
>> Local RAFT log contains all the DB changes that are not part of a
>> snapshot yet. Since snapshots are taken at most once in 10 minutes,
>> log could grow pretty big. Up to tens of thousands of entries and
>> each of these entries could be fairly big by themselves. That being
>> said RAFT log could grow up to tens of GBs too.
>>
>> One extra point for memory consumption is that memory likely doesn't
>> go away even after calling free() due to implementation of a C memory
>> allocators. And this happens a lot. ovsdb-server process usually
>> holds a lot of system memory even if the database is almost empty.
>> This heap memory might be returned back to OS by using malloc_trim().
>>
>> --
>> All of these issues was found on branch-2.13, but it always hard to
>> distinguish new features from the bug fix when we're talking about
>> scaling issues. Anyway, I think, it'll be good to have these
>> patches (if they are any good) backorted to 2.13, especially because
>> it's going to be our next LTS. Thoughts?
>>
>
> Hi Ilya,
>
> I think although these might be considered features, without them there doesn't
> seem to be a way to address the memory consumption issues in production
> deployments.
>
> In my opinion, these should definitely go to 2.13 branch too.
OK. Thanks!
I'll backport them.
>
> Thanks,
> Dumitru
>
>> Ilya Maximets (5):
>> raft: Add log length to the memory report.
>> ovsdb-server: Reclaim heap memory after compaction.
>> raft: Set threshold on backlog for raft connections.
>> raft: Make backlog thresholds configurable.
>> raft: Avoid having more than one snapshot in-flight.
>>
>> NEWS | 6 +++
>> configure.ac | 1 +
>> lib/jsonrpc.c | 57 ++++++++++++++++++++++++-
>> lib/jsonrpc.h | 6 +++
>> ovsdb/ovsdb-server.1.in | 9 ++++
>> ovsdb/ovsdb-server.c | 41 +++++++++++++++++-
>> ovsdb/ovsdb.c | 12 +++++-
>> ovsdb/ovsdb.h | 3 +-
>> ovsdb/raft-private.c | 1 -
>> ovsdb/raft-private.h | 4 +-
>> ovsdb/raft.c | 93 +++++++++++++++++++++++++++++------------
>> 11 files changed, 199 insertions(+), 34 deletions(-)
>>
>
More information about the dev
mailing list