[ovs-discuss] [OVN] Cluster mode ovsdb memory keeps increasing

刘梦馨 liumengxinfly at gmail.com
Mon Dec 16 03:54:14 UTC 2019


Hi, Han

In my test scenario, I use ovn-ctl to start a one node ovn with cluster
mode db and no chassis bind to the ovn-sb to just check the memory usage of
ovn-nb.
Then use a script to add a logical switch, add 1000 ports, set dynamic
addresses and then delete the logical switch.

#!/bin/bash
ovn-nbctl ls-add ls1
for i in {1..1000}; do
  ovn-nbctl lsp-add ls1 ls1-vm$i
  ovn-nbctl lsp-set-addresses ls1-vm$i dynamic
done
ovn-nbctl ls-del ls1

I run this script repeatedly and watch the memory change.

After 5 runs (5000 lsp add and delete), the rss of nb increased to 667M.
The nb file increased to 119M and didn't automatically compacted. After a
manually compact the db file size change back to 11K, but the memory usage
didn't change.



Han Zhou <hzhou at ovn.org> 于2019年12月14日周六 上午3:40写道:

>
>
> On Wed, Dec 11, 2019 at 12:51 AM 刘梦馨 <liumengxinfly at gmail.com> wrote:
> >
> >
> > We are using ovs/ovn 2.12.0 to implementing our container network. After
> switching form standalone ovndb to cluster mode ovndb, we noticed that the
> memory consumption for both ovnnb and ovnsb will keep increasing after each
> operation and never decrease.
> >
> > We did some profiling by valgrind. The leak check report a 16 byte leak
> in fork_and_wait_for_startup, which obviously is not the main reason. Later
> we use memif to profile the memory consumption and we put the result in the
> attachment.
> >
> > Most of the memory come from two part ovsthread_wrapper
> (ovs-thread.c:378) that allocates a subprogram_name and  jsonrpc_send
> (jsonrpc.c:253) as below, (I just skipped the duplicated stack of jsonrpc).
> >
> > However I found both part have a related free operation in near place,
> so I don't know how to further explore this memory issue. I'm not aware of
> the differences here between cluster mode and standalone mode.
> >
> > Can anyone give some advice and hint? Thanks!
> >
> > 100.00% (357,920,768B) (page allocation syscalls) mmap/mremap/brk,
> --alloc-fns, etc.
> > ->78.52% (281,038,848B) 0x66FDD49: mmap (in /usr/lib64/libc-2.17.so)
> > | ->37.50% (134,217,728B) 0x66841EF: new_heap (in /usr/lib64/
> libc-2.17.so)
> > | | ->37.50% (134,217,728B) 0x6684C22: arena_get2.isra.3 (in /usr/lib64/
> libc-2.17.so)
> > | |   ->37.50% (134,217,728B) 0x668AACC: malloc (in /usr/lib64/
> libc-2.17.so)
> > | |     ->37.50% (134,217,728B) 0x4FDC613: xmalloc (util.c:138)
> > | |       ->37.50% (134,217,728B) 0x4FDC78E: xvasprintf (util.c:202)
> > | |         ->37.50% (134,217,728B) 0x4FDC877: xasprintf (util.c:343)
> > | |           ->37.50% (134,217,728B) 0x4FA548D: ovsthread_wrapper
> (ovs-thread.c:378)
> > | |             ->37.50% (134,217,728B) 0x5BE5E63: start_thread (in
> /usr/lib64/libpthread-2.17.so)
> > | |               ->37.50% (134,217,728B) 0x670388B: clone (in
> /usr/lib64/libc-2.17.so)
> > | |
> > | ->36.33% (130,023,424B) 0x6686DF3: sysmalloc (in /usr/lib64/
> libc-2.17.so)
> > | | ->36.33% (130,023,424B) 0x6687CA8: _int_malloc (in /usr/lib64/
> libc-2.17.so)
> > | |   ->28.42% (101,711,872B) 0x66890C0: _int_realloc (in /usr/lib64/
> libc-2.17.so)
> > | |   | ->28.42% (101,711,872B) 0x668B160: realloc (in /usr/lib64/
> libc-2.17.so)
> > | |   |   ->28.42% (101,711,872B) 0x4FDC9A3: xrealloc (util.c:149)
> > | |   |     ->28.42% (101,711,872B) 0x4F1DEB2: ds_reserve
> (dynamic-string.c:63)
> > | |   |       ->28.42% (101,711,872B) 0x4F1DED3: ds_put_uninit
> (dynamic-string.c:73)
> > | |   |         ->28.42% (101,711,872B) 0x4F1DF0B: ds_put_char__
> (dynamic-string.c:82)
> > | |   |           ->26.37% (94,371,840B) 0x4F2B09F:
> json_serialize_string (dynamic-string.h:93)
> > | |   |           | ->12.01% (42,991,616B) 0x4F2B3EA: json_serialize
> (json.c:1651)
> > | |   |           | | ->12.01% (42,991,616B) 0x4F2B3EA: json_serialize
> (json.c:1651)
> > | |   |           | |   ->12.01% (42,991,616B) 0x4F2B3EA: json_serialize
> (json.c:1651)
> > | |   |           | |     ->12.01% (42,991,616B) 0x4F2B540:
> json_serialize (json.c:1626)
> > | |   |           | |       ->12.01% (42,991,616B) 0x4F2B540:
> json_serialize (json.c:1626)
> > | |   |           | |         ->12.01% (42,991,616B) 0x4F2B540:
> json_serialize (json.c:1626)
> > | |   |           | |           ->12.01% (42,991,616B) 0x4F2B540:
> json_serialize (json.c:1626)
> > | |   |           | |             ->12.01% (42,991,616B) 0x4F2B3EA:
> json_serialize (json.c:1651)
> > | |   |           | |               ->12.01% (42,991,616B) 0x4F2B540:
> json_serialize (json.c:1626)
> > | |   |           | |                 ->12.01% (42,991,616B) 0x4F2D82A:
> json_to_ds (json.c:1525)
> > | |   |           | |                   ->12.01% (42,991,616B)
> 0x4F2EA49: jsonrpc_send (jsonrpc.c:253)
> > | |   |           | |                     ->12.01% (42,991,616B)
> 0x4C3A68A: ovsdb_jsonrpc_server_run (jsonrpc-server.c:1104)
> > | |   |           | |                       ->12.01% (42,991,616B)
> 0x10DCC1: main (ovsdb-server.c:209)
> >
> > _______________________________________________
> > discuss mailing list
> > discuss at openvswitch.org
> > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
>
> Thanks for reporting the issue. Could you describe your test scenario (the
> operations), the scale, the db file size and the memory (RSS) data of the
> NB/SB?
> Clustered mode maintains some extra data such as RAFT logs, compares to
> standalone, but it should not increase forever, because RAFT logs will get
> compacted periodically.
>
> Thanks,
> Han
>


-- 
刘梦馨
Blog: http://oilbeater.com
Weibo: @oilbeater <http://weibo.com/oilbeater>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20191216/4943853a/attachment.html>


More information about the discuss mailing list