[ovs-dev] ovn-northd-ddlog - high mem and cpu usage when started with an existing DB

Dumitru Ceara dceara at redhat.com
Thu Jul 8 18:59:24 UTC 2021


Hi Ben,

As discussed earlier, during the OVN meeting, I've noticed a new
performance issue with ovn-northd-ddlog when running it against a
database from one of our more recent scale tests:

http://people.redhat.com/~dceara/ovn-northd-ddlog-tests/20210708/ovnnb_db.db

ovn-northd-ddlog uses 100% CPU and never really reaches the point to
perform the first transaction to the Southbound.  Memory usage is also
very high, I stopped it at 45GB RSS.

To test I did:
SANDBOXFLAGS="--nbdb-source=/tmp/ovnnb_db.db --ddlog" make sandbox

(gdb) thread 4
[Switching to thread 4 (Thread 0x7fc079ab7700 (LWP 2440622))]
#0  0x00007fc07a361158 in _int_malloc (av=av at entry=0x7fc070000020, bytes=bytes at entry=33) at malloc.c:3862
3862                            fwd = fwd->fd;
(gdb) bt
#0  0x00007fc07a361158 in _int_malloc (av=av at entry=0x7fc070000020, bytes=bytes at entry=33) at malloc.c:3862
#1  0x00007fc07a36196e in _int_realloc (av=av at entry=0x7fc070000020, oldp=oldp at entry=0x7fb8c4f4ff30, oldsize=oldsize at entry=32, nb=48) at malloc.c:4600
#2  0x00007fc07a362d86 in __GI___libc_realloc (oldmem=0x7fb8c4f4ff40, bytes=32) at malloc.c:3235
#3  0x00000000014ca970 in alloc::raw_vec::finish_grow ()
#4  0x00000000014caa6d in alloc::raw_vec::RawVec<T,A>::reserve ()
#5  0x0000000000e511a3 in ddlog_rt::string_append_str ()
#6  0x000000000079162f in types::__Rule_Flow_80::{{closure}}::__f ()
#7  0x00000000010ec3e8 in differential_dataflow::operators::join::Deferred<K,V1,V2,T,R1,R2,R3,C1,C2,M,D>::work::{{closure}} ()
#8  0x000000000108b924 in differential_dataflow::operators::join::Deferred<K,V1,V2,T,R1,R2,R3,C1,C2,M,D>::work ()
#9  0x0000000001a07adf in <timely::dataflow::operators::generic::builder_raw::OperatorCore<T,L> as timely::scheduling::Schedule>::schedule ()
#10 0x00000000010258cd in <timely::progress::subgraph::Subgraph<TOuter,TInner> as timely::scheduling::Schedule>::schedule ()
#11 0x00000000011da3da in timely::worker::Wrapper::step ()
#12 0x000000000164f04f in timely::worker::Worker<A>::step_or_park ()
#13 0x00000000017b971f in differential_datalog::program::worker::DDlogWorker::flush ()
#14 0x00000000017b8682 in differential_datalog::program::worker::DDlogWorker::run ()
#15 0x000000000157b22a in std::sys_common::backtrace::__rust_begin_short_backtrace ()
#16 0x0000000000ff6a37 in core::ops::function::FnOnce::call_once{{vtable-shim}} ()
#17 0x0000000001265dfa in std::sys::unix::thread::Thread::new::thread_start ()
#18 0x00007fc07a767432 in start_thread (arg=<optimized out>) at pthread_create.c:477
#19 0x00007fc07a3d76d3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Regards,
Dumitru



More information about the dev mailing list