[ovs-discuss] [OVN] Cluster mode ovsdb memory keeps increasing

Han Zhou hzhou at ovn.org
Fri Dec 13 19:40:05 UTC 2019


On Wed, Dec 11, 2019 at 12:51 AM 刘梦馨 <liumengxinfly at gmail.com> wrote:
>
>
> We are using ovs/ovn 2.12.0 to implementing our container network. After
switching form standalone ovndb to cluster mode ovndb, we noticed that the
memory consumption for both ovnnb and ovnsb will keep increasing after each
operation and never decrease.
>
> We did some profiling by valgrind. The leak check report a 16 byte leak
in fork_and_wait_for_startup, which obviously is not the main reason. Later
we use memif to profile the memory consumption and we put the result in the
attachment.
>
> Most of the memory come from two part ovsthread_wrapper
(ovs-thread.c:378) that allocates a subprogram_name and  jsonrpc_send
(jsonrpc.c:253) as below, (I just skipped the duplicated stack of jsonrpc).
>
> However I found both part have a related free operation in near place, so
I don't know how to further explore this memory issue. I'm not aware of the
differences here between cluster mode and standalone mode.
>
> Can anyone give some advice and hint? Thanks!
>
> 100.00% (357,920,768B) (page allocation syscalls) mmap/mremap/brk,
--alloc-fns, etc.
> ->78.52% (281,038,848B) 0x66FDD49: mmap (in /usr/lib64/libc-2.17.so)
> | ->37.50% (134,217,728B) 0x66841EF: new_heap (in /usr/lib64/libc-2.17.so)
> | | ->37.50% (134,217,728B) 0x6684C22: arena_get2.isra.3 (in /usr/lib64/
libc-2.17.so)
> | |   ->37.50% (134,217,728B) 0x668AACC: malloc (in /usr/lib64/
libc-2.17.so)
> | |     ->37.50% (134,217,728B) 0x4FDC613: xmalloc (util.c:138)
> | |       ->37.50% (134,217,728B) 0x4FDC78E: xvasprintf (util.c:202)
> | |         ->37.50% (134,217,728B) 0x4FDC877: xasprintf (util.c:343)
> | |           ->37.50% (134,217,728B) 0x4FA548D: ovsthread_wrapper
(ovs-thread.c:378)
> | |             ->37.50% (134,217,728B) 0x5BE5E63: start_thread (in
/usr/lib64/libpthread-2.17.so)
> | |               ->37.50% (134,217,728B) 0x670388B: clone (in /usr/lib64/
libc-2.17.so)
> | |
> | ->36.33% (130,023,424B) 0x6686DF3: sysmalloc (in /usr/lib64/libc-2.17.so
)
> | | ->36.33% (130,023,424B) 0x6687CA8: _int_malloc (in /usr/lib64/
libc-2.17.so)
> | |   ->28.42% (101,711,872B) 0x66890C0: _int_realloc (in /usr/lib64/
libc-2.17.so)
> | |   | ->28.42% (101,711,872B) 0x668B160: realloc (in /usr/lib64/
libc-2.17.so)
> | |   |   ->28.42% (101,711,872B) 0x4FDC9A3: xrealloc (util.c:149)
> | |   |     ->28.42% (101,711,872B) 0x4F1DEB2: ds_reserve
(dynamic-string.c:63)
> | |   |       ->28.42% (101,711,872B) 0x4F1DED3: ds_put_uninit
(dynamic-string.c:73)
> | |   |         ->28.42% (101,711,872B) 0x4F1DF0B: ds_put_char__
(dynamic-string.c:82)
> | |   |           ->26.37% (94,371,840B) 0x4F2B09F: json_serialize_string
(dynamic-string.h:93)
> | |   |           | ->12.01% (42,991,616B) 0x4F2B3EA: json_serialize
(json.c:1651)
> | |   |           | | ->12.01% (42,991,616B) 0x4F2B3EA: json_serialize
(json.c:1651)
> | |   |           | |   ->12.01% (42,991,616B) 0x4F2B3EA: json_serialize
(json.c:1651)
> | |   |           | |     ->12.01% (42,991,616B) 0x4F2B540:
json_serialize (json.c:1626)
> | |   |           | |       ->12.01% (42,991,616B) 0x4F2B540:
json_serialize (json.c:1626)
> | |   |           | |         ->12.01% (42,991,616B) 0x4F2B540:
json_serialize (json.c:1626)
> | |   |           | |           ->12.01% (42,991,616B) 0x4F2B540:
json_serialize (json.c:1626)
> | |   |           | |             ->12.01% (42,991,616B) 0x4F2B3EA:
json_serialize (json.c:1651)
> | |   |           | |               ->12.01% (42,991,616B) 0x4F2B540:
json_serialize (json.c:1626)
> | |   |           | |                 ->12.01% (42,991,616B) 0x4F2D82A:
json_to_ds (json.c:1525)
> | |   |           | |                   ->12.01% (42,991,616B) 0x4F2EA49:
jsonrpc_send (jsonrpc.c:253)
> | |   |           | |                     ->12.01% (42,991,616B)
0x4C3A68A: ovsdb_jsonrpc_server_run (jsonrpc-server.c:1104)
> | |   |           | |                       ->12.01% (42,991,616B)
0x10DCC1: main (ovsdb-server.c:209)
>
> _______________________________________________
> discuss mailing list
> discuss at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss

Thanks for reporting the issue. Could you describe your test scenario (the
operations), the scale, the db file size and the memory (RSS) data of the
NB/SB?
Clustered mode maintains some extra data such as RAFT logs, compares to
standalone, but it should not increase forever, because RAFT logs will get
compacted periodically.

Thanks,
Han
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20191213/72327feb/attachment.html>


More information about the discuss mailing list