[ovs-discuss] OVN Database sizes - Auto compact feature

Daniel Alvarez Sanchez dalvarez at redhat.com
Wed Mar 7 16:53:15 UTC 2018


Repeated the test with 1000 ports this time. See attached image.
For some reason, the sizes grow while deleting the ports (the
deletion task starts at around x=2500). The weird thing is why
they keep growing and the online compact doesn't work as when
I do it through ovs-appctl tool.

I suspect this is a bug and eventually it will grow and grow unless
we manually compact the db.

NB database:
2018-03-07T13:50:33.671Z|00058|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519121698.545 seconds old, 69 transactions)
2018-03-07T14:38:35.518Z|00059|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519118104.064 seconds old, 436 transactions)
2018-03-07T14:48:35.589Z|00060|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115822.254 seconds old, 386 transactions)
2018-03-07T14:58:37.155Z|00061|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115823.488 seconds old, 353 transactions)
2018-03-07T15:08:39.030Z|00062|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115823.975 seconds old, 415 transactions)
2018-03-07T15:18:40.348Z|00063|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115823.413 seconds old, 393 transactions)
2018-03-07T15:28:45.405Z|00064|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115827.054 seconds old, 141 transactions)
2018-03-07T15:38:52.894Z|00082|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115825.353 seconds old, 134 transactions)
2018-03-07T15:48:54.079Z|00083|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115823.297 seconds old, 155 transactions)
2018-03-07T15:58:57.185Z|00084|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115825.234 seconds old, 177 transactions)
2018-03-07T16:08:58.955Z|00085|ovsdb_file|INFO|/opt/stack/data/ovs/ovnnb_db.db:
compacting database online (1519115823.913 seconds old, 213 transactions)

SB database:
2018-03-07T13:32:21.672Z|00022|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (1519124364.908 seconds old, 951 transactions)
2018-03-07T14:47:40.293Z|00023|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (1519119740.823 seconds old, 776 transactions)
2018-03-07T15:05:10.797Z|00024|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (1519116272.507 seconds old, 666 transactions)
2018-03-07T15:33:49.467Z|00025|ovsdb_file|INFO|/opt/stack/data/ovs/ovnsb_db.db:
compacting database online (1519116940.094 seconds old, 748 transactions)

When compacting the DB's manually they shrink a lot:

[stack at ovn ovs]$ ls -alh ovn*.db
-rw-r--r--. 1 stack stack 9.5M Mar  7 16:17 ovnnb_db.db
-rw-r--r--. 1 stack stack  32M Mar  7 16:17 ovnsb_db.db
[stack at ovn ovs]$ sudo ovs-appctl -t
/usr/local/var/run/openvswitch/ovnnb_db.ctl ovsdb-server/compact


[stack at ovn ovs]$ sudo ovs-appctl -t
/usr/local/var/run/openvswitch/ovnsb_db.ctl ovsdb-server/compact
[stack at ovn ovs]$ ls -alh ovn*.db
-rw-r--r--. 1 stack stack  39K Mar  7 16:52 ovnnb_db.db
-rw-r--r--. 1 stack stack 207K Mar  7 16:52 ovnsb_db.db



On Wed, Mar 7, 2018 at 3:42 PM, Daniel Alvarez Sanchez <dalvarez at redhat.com>
wrote:

> Right, thanks Mark! Good point about the 4x, I missed that one.
> I'm repeating the test for 1K ports. Not sure if I'll be able to reproduce
> the 2.5GB part but it is still weird that while deleting ports (actually
> deleting stuff from the DB) it doesn't get to compact the DB further
> (last time it shrinked to 9MB) so maybe we have something odd here
> going in the online compact. I'll post the results after this test.
>
> On Wed, Mar 7, 2018 at 3:35 PM, Mark Michelson <mmichels at redhat.com>
> wrote:
>
>> On 03/07/2018 07:40 AM, Daniel Alvarez Sanchez wrote:
>>
>>> Hi folks,
>>>
>>> During the performance tests I've been doing lately I noticed
>>> that the size of the Southbound database was around 2.5GB
>>> in one of my setups. I couldn't dig further then but now I
>>> decided to explore a bit more and these are the results in
>>> my all-in-one OpenStack setup using OVN as a backend:
>>>
>>> * Created 800 ports on the same network (logical switch).
>>> * Deleted those 800 ports.
>>> * I logged the DB sizes for both NB and SB databases every second.
>>>
>>> See attached image for the results.
>>>
>>> At around x=2000, the creation task finished and deletion starts.
>>> As you can see, there's automatic compact happening in the
>>> NB database across the whole test. However, while I was deleting
>>> ports, the SB database stop shrinking and keeps growing.
>>>
>>> After the test finished, the DB sizes remaining the same
>>> thus SB database being around 34MB. It was not until I
>>> manually compacted it when it finally shrinked:
>>>
>>>
>>> [stack at ovn ovs]$ ls -alh ovnsb_db.db
>>> -rw-r--r--. 1 stack stack 34M Mar  7 12:04 ovnsb_db.db
>>>
>>> [stack at ovn ovs]$ sudo ovs-appctl -t /usr/local/var/run/openvswitch/ovnsb_db.ctl
>>> ovsdb-server/compact
>>>
>>> [stack at ovn ovs]$ ls -alh ovnsb_db.db
>>> -rw-r--r--. 1 stack stack 207K Mar  7 13:32 ovnsb_db.db
>>>
>>> I'll try to investigate further in the code.
>>> Thanks,
>>>
>>> Daniel
>>>
>>
>> Daniel and I discussed this offline and I think the attached result does
>> not necessarily indicate a bug yet.
>>
>> One of the requirements for the DB to compact automatically is that it
>> needs to grow to at least 4x the size of the previous snapshot. In the
>> graph, we can see that the first time the SB DB compacts (around x==1000),
>> it shrinks down to ~5MB. We would expect the DB to compact again when it
>> gets to ~20MB. This happens at around x==1900. The DB shrinks to ~9MB, so
>> we would expect it to shrink again when it reaches ~36MB. When the test
>> ends, the SB DB is ~34 MB, so it is not quite large enough to compact yet.
>> If the test had gone a bit longer, then presumably we might have seen the
>> DB shrink again.
>>
>> It will be interesting to see the test that leads to a 2.5GB southbound
>> database.
>>
>> Mark!
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20180307/0ddec8a1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ovn_db_sizes_1K.png
Type: image/png
Size: 101270 bytes
Desc: not available
URL: <http://mail.openvswitch.org/pipermail/ovs-discuss/attachments/20180307/0ddec8a1/attachment-0001.png>


More information about the discuss mailing list