[ovs-dev] ovn-northd-ddlog scale issues
Ben Pfaff
blp at ovn.org
Wed Mar 24 15:48:38 UTC 2021
Thanks a lot for the report! I need a new test case so I'll give this
one a shot when I can.
On Wed, Mar 24, 2021 at 04:03:07PM +0100, Dumitru Ceara wrote:
> Hi Ben,
>
> We discussed a bit about this during one of the recent IRC OVN meetings,
> but I didn't get around to properly reporting this until now.
>
> I've tried running ovn-northd-ddlog against some large OVN NB/DB
> databases extracted from one of our scale testing runs:
>
> http://people.redhat.com/~dceara/ovn-northd-ddlog-tests/20210324/existing-nb-sb/
>
> It seems that ovn-northd-ddlog gets stuck in a busy loop and uses a lot
> of memory:
>
> 775734 root 10 -10 81.6g 80.8g 22396 S 99.7 64.2 3:50.79 ovn-northd-ddlog
>
> ovn-northd-ddlog is stuck in an (infinite?) loop in
> ddlog_transaction_commit_dump_changes():
>
> (gdb) bt
> #0 0x00007f2762167e92 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
> #1 0x0000000000891eb3 in std::thread::park ()
> #2 0x0000000000530963 in crossbeam_channel::context::Context::wait_until ()
> #3 0x0000000000530aa1 in crossbeam_channel::context::Context::with::{{closure}} ()
> #4 0x0000000000531f5c in crossbeam_channel::flavors::list::Channel<T>::recv ()
> #5 0x000000000075b35d in crossbeam_channel::channel::Receiver<T>::recv ()
> #6 0x0000000000512156 in differential_datalog::program::RunningProgram::flush ()
> #7 0x000000000050de67 in differential_datalog::program::RunningProgram::transaction_commit ()
> #8 0x000000000093f17d in <ovn_northd_ddlog::api::HDDlog as differential_datalog::ddlog::DDlog>::transaction_commit_dump_changes ()
> #9 0x000000000040cc40 in ddlog_transaction_commit_dump_changes ()
> #10 0x000000000040a758 in ddlog_commit (ddlog=<optimized out>) at northd/ovn-northd-ddlog.c:435
> #11 northd_parse_update (update=0x3dc0598, ctx=0x3d71880) at northd/ovn-northd-ddlog.c:435
> #12 northd_run (ctx=0x3d71880) at northd/ovn-northd-ddlog.c:512
> #13 0x0000000000408edd in main (argc=<optimized out>, argv=<optimized out>) at northd/ovn-northd-ddlog.c:1203
>
> For comparison, running the C version of ovn-northd I get:
>
> 2021-03-24T14:48:06.556Z|00033|timeval|WARN|Unreasonably long 11290ms poll interval (10916ms user, 334ms system)
> 2021-03-24T14:48:14.678Z|00050|timeval|WARN|Unreasonably long 8122ms poll interval (8046ms user, 48ms system)
>
> But after the northd iteration ends, memory usage is OK:
>
> 777567 root 10 -10 1657308 1.6g 3496 S 0.0 1.2 0:49.08 ovn-northd
>
> The behavior above is consistent both with current OVN master and also
> when cherry-picking the ddlog-related patches that are pending in
> patchwork:
> http://patchwork.ozlabs.org/project/ovn/list/?series=233075
> http://patchwork.ozlabs.org/project/ovn/list/?series=232480
> http://patchwork.ozlabs.org/project/ovn/list/?series=233079
> http://patchwork.ozlabs.org/project/ovn/list/?series=233080
>
> I didn't try out the changes from the following series though as I
> understand they need a v2:
> http://patchwork.ozlabs.org/project/ovn/list/?series=232040
>
> Regards,
> Dumitru
>
More information about the dev
mailing list