[ovs-dev] ovn-northd-ddlog scale issues

Dumitru Ceara dceara at redhat.com
Mon Jun 28 15:40:53 UTC 2021


On 5/20/21 5:50 PM, Ben Pfaff wrote:
> On Thu, May 20, 2021 at 05:06:26PM +0200, Dumitru Ceara wrote:
>> On 4/7/21 6:49 PM, Ben Pfaff wrote:
>>
>> [...]
>>
>>>>
>>>> Thanks!  I can download them now.  It's back on my to-do list.
>>>
>>> I can reproduce the problem now.  I haven't fixed it yet, but I did fix
>>> a nasty performance problem in ovn-nbctl that became really apparent
>>> when working with your databases:
>>> https://mail.openvswitch.org/pipermail/ovs-dev/2021-April/381909.html
>>
>> I was wondering if you had a chance to look at this since.
> 
> I haven't kept going.  I consider my series that gives a 5x performance
> improvement a kind of checkpoint along the way.  I assumed at first that
> it would get reviewed quickly so I could move on to other things, but no
> one has looked at it yet.
> 


Hi Ben,

Just a note, I've tried this again with ovn-northd-ddlog built from
current OVN master branch, running against the same DBs:

http://people.redhat.com/~dceara/ovn-northd-ddlog-tests/20210324/existing-nb-sb/

I don't see the huge memory usage as initially reported, now
ovn-northd-ddlog is stable at ~5GB, but it still seems that
ovn-northd-ddlog spins indeterminately (infinite loop?) when starting up.

Attaching GDB to it I see:

(gdb) bt
#0  syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1  0x00000000011bd9fa in std::thread::park ()
#2  0x0000000000f824e3 in crossbeam_channel::context::Context::wait_until ()
#3  0x0000000000f82621 in crossbeam_channel::context::Context::with::{{closure}} ()
#4  0x0000000000f83468 in crossbeam_channel::flavors::list::Channel<T>::recv ()
#5  0x000000000149b60d in crossbeam_channel::channel::Receiver<T>::recv ()
#6  0x0000000000f4f2d0 in differential_datalog::program::RunningProgram::await_flush_ack ()
#7  0x0000000000f4af4f in differential_datalog::program::RunningProgram::transaction_commit ()
#8  0x0000000000507610 in <ovn_northd_ddlog::api::HDDlog as differential_datalog::ddlog::DDlog>::transaction_commit_dump_changes ()
#9  0x0000000000528bb0 in ddlog_transaction_commit_dump_changes ()
#10 0x000000000040a874 in ddlog_commit (delta=0x44ba7e0, ddlog=<optimized out>) at northd/ovn-northd-ddlog.c:335
#11 northd_parse_updates (updates=0x7ffd77a00aa0, ctx=0x444e690) at northd/ovn-northd-ddlog.c:464
#12 northd_run (ctx=0x444e690) at northd/ovn-northd-ddlog.c:566
#13 0x0000000000408efc in main (argc=<optimized out>, argv=<optimized out>) at northd/ovn-northd-ddlog.c:1296

Regards,
Dumitru



More information about the dev mailing list