[ovs-dev] ovn-northd-ddlog scale issues

Ben Pfaff blp at ovn.org
Wed Mar 31 20:12:04 UTC 2021


I found the leaks and the fixes will be in v2 of the series.

On Thu, Mar 25, 2021 at 01:14:37PM -0700, Ben Pfaff wrote:
> OK.  I guess I'll have to try it myself.
> 
> On Thu, Mar 25, 2021 at 12:59:00PM +0530, Numan Siddique wrote:
> > Hi Ben,
> > 
> > With your memory fixes applied, I still see many test failures due to
> > memory leaks
> > when configured as:
> > 
> > ./configure --enable-Werror --enable-sparse CFLAGS="-g -fsanitize=address"
> > 
> > You can find more details about the memory leaks here -
> > https://gist.github.com/numansiddique/e701777da977a89e24b49f159bb30d5d
> > 
> > Thanks
> > Numan
> > 
> > 
> > 
> > On Wed, Mar 24, 2021 at 9:19 PM Ben Pfaff <blp at ovn.org> wrote:
> > >
> > > Thanks a lot for the report!  I need a new test case so I'll give this
> > > one a shot when I can.
> > >
> > > On Wed, Mar 24, 2021 at 04:03:07PM +0100, Dumitru Ceara wrote:
> > > > Hi Ben,
> > > >
> > > > We discussed a bit about this during one of the recent IRC OVN meetings,
> > > > but I didn't get around to properly reporting this until now.
> > > >
> > > > I've tried running ovn-northd-ddlog against some large OVN NB/DB
> > > > databases extracted from one of our scale testing runs:
> > > >
> > > > http://people.redhat.com/~dceara/ovn-northd-ddlog-tests/20210324/existing-nb-sb/
> > > >
> > > > It seems that ovn-northd-ddlog gets stuck in a busy loop and uses a lot
> > > > of memory:
> > > >
> > > > 775734 root      10 -10   81.6g  80.8g  22396 S  99.7  64.2   3:50.79 ovn-northd-ddlog
> > > >
> > > > ovn-northd-ddlog is stuck in an (infinite?) loop in
> > > > ddlog_transaction_commit_dump_changes():
> > > >
> > > > (gdb) bt
> > > > #0  0x00007f2762167e92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> > > > #1  0x0000000000891eb3 in std::thread::park ()
> > > > #2  0x0000000000530963 in crossbeam_channel::context::Context::wait_until ()
> > > > #3  0x0000000000530aa1 in crossbeam_channel::context::Context::with::{{closure}} ()
> > > > #4  0x0000000000531f5c in crossbeam_channel::flavors::list::Channel<T>::recv ()
> > > > #5  0x000000000075b35d in crossbeam_channel::channel::Receiver<T>::recv ()
> > > > #6  0x0000000000512156 in differential_datalog::program::RunningProgram::flush ()
> > > > #7  0x000000000050de67 in differential_datalog::program::RunningProgram::transaction_commit ()
> > > > #8  0x000000000093f17d in <ovn_northd_ddlog::api::HDDlog as differential_datalog::ddlog::DDlog>::transaction_commit_dump_changes ()
> > > > #9  0x000000000040cc40 in ddlog_transaction_commit_dump_changes ()
> > > > #10 0x000000000040a758 in ddlog_commit (ddlog=<optimized out>) at northd/ovn-northd-ddlog.c:435
> > > > #11 northd_parse_update (update=0x3dc0598, ctx=0x3d71880) at northd/ovn-northd-ddlog.c:435
> > > > #12 northd_run (ctx=0x3d71880) at northd/ovn-northd-ddlog.c:512
> > > > #13 0x0000000000408edd in main (argc=<optimized out>, argv=<optimized out>) at northd/ovn-northd-ddlog.c:1203
> > > >
> > > > For comparison, running the C version of ovn-northd I get:
> > > >
> > > > 2021-03-24T14:48:06.556Z|00033|timeval|WARN|Unreasonably long 11290ms poll interval (10916ms user, 334ms system)
> > > > 2021-03-24T14:48:14.678Z|00050|timeval|WARN|Unreasonably long 8122ms poll interval (8046ms user, 48ms system)
> > > >
> > > > But after the northd iteration ends, memory usage is OK:
> > > >
> > > > 777567 root      10 -10 1657308   1.6g   3496 S   0.0   1.2   0:49.08 ovn-northd
> > > >
> > > > The behavior above is consistent both with current OVN master and also
> > > > when cherry-picking the ddlog-related patches that are pending in
> > > > patchwork:
> > > > http://patchwork.ozlabs.org/project/ovn/list/?series=233075
> > > > http://patchwork.ozlabs.org/project/ovn/list/?series=232480
> > > > http://patchwork.ozlabs.org/project/ovn/list/?series=233079
> > > > http://patchwork.ozlabs.org/project/ovn/list/?series=233080
> > > >
> > > > I didn't try out the changes from the following series though as I
> > > > understand they need a v2:
> > > > http://patchwork.ozlabs.org/project/ovn/list/?series=232040
> > > >
> > > > Regards,
> > > > Dumitru
> > > >
> > > _______________________________________________
> > > dev mailing list
> > > dev at openvswitch.org
> > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev
> > >


More information about the dev mailing list