[ovs-dev] [PATCH ovn v3 3/3] tests: Add check-perf target

Dumitru Ceara dceara at redhat.com
Wed Jun 30 13:46:55 UTC 2021


On 6/30/21 2:49 PM, Mark Gray wrote:
> On 24/06/2021 16:34, Dumitru Ceara wrote:
>> On 6/18/21 10:52 AM, Mark Gray wrote:
>>> Add a suite of micro-benchmarks to aid a developer in understanding the
>>> performance impact of any changes that they are making. They can be used to
>>> help to understand the relative performance between two test runs on the same
>>> test machine, but are not intended to give the absolute performance of OVN.
>>>
>>> To invoke the performance testsuite, run:
>>>
>>>     $ make check-perf
>>>
>>> This will run all available performance tests.
>>>
>>> Additional metrics (e.g. memory, coverage, perf counters) may be added
>>> in the future. Additional tests (e.g. additional topologies,  ovn-controller
>>> tests) may be added in the future.
>>>
>>> Signed-off-by: Mark Gray <mark.d.gray at redhat.com>
>>> ---
>>
>> Thanks for this, I think this is a very good idea!
> 
> Thanks (and thanks for the review!), I think it is a good starting part
> but it will only be useful if it gets used! Personally, I can see myself
> using this. However, if it doesn't, it is not very invasive and could
> easily be let rot or removed.
>>
>>>
>>> Notes:
>>>     v2:  create results directory to fix build error
>>>     v3:  forgot to commit, create results directory to fix build error
>>>
>>>  Documentation/topics/testing.rst |  49 ++++++++
>>>  tests/.gitignore                 |   3 +
>>>  tests/automake.mk                |  27 ++++
>>>  tests/perf-northd.at             | 207 +++++++++++++++++++++++++++++++
>>>  tests/perf-testsuite.at          |  26 ++++
>>>  5 files changed, 312 insertions(+)
>>>  create mode 100644 tests/perf-northd.at
>>>  create mode 100644 tests/perf-testsuite.at
>>>
>>> diff --git a/Documentation/topics/testing.rst b/Documentation/topics/testing.rst
>>> index be9e7c57331c..ccd3278437b1 100644
>>> --- a/Documentation/topics/testing.rst
>>> +++ b/Documentation/topics/testing.rst
>>> @@ -256,3 +256,52 @@ the following::
>>>  All the features documented under `Unit Tests`_ are available for the
>>>  datapath testsuites, except that the datapath testsuites do not
>>>  support running tests in parallel.
>>> +
>>> +Performance testing
>>> +~~~~~~~~~~~~~~~~~~~
>>> +
>>> +OVN includes a suite of micro-benchmarks to aid a developer in understanding the
>>> +performance impact of any changes that they are making. They can be used to
>>> +help to understand the relative performance between two test runs on the same
>>> +test machine, but are not intended to give the absolute performance of OVN.
>>> +
>>> +To invoke the performance testsuite, run::
>>> +
>>> +    $ make check-perf
>>> +
>>> +This will run all available performance tests. Some of these tests may be
>>> +long-running as they need to build complex logical network topologies. In order
>>> +to speed up subsequent test runs, some objects (e.g. the Northbound DB) may be
>>> +cached. In order to force the tests to rebuild all these objects, run::
>>> +
>>> +    $ make check-perf TESTSUITEFLAGS="--rebuild"
>>> +
>>> +A typical workflow for a developer trying to improve the performance of OVN
>>> +would be the following:
>>> +
>>> +0. Optional: Modify/add a performance test to buld the topology that you are
>>> +   benchmarking, if required.
>>> +1. Run ``make check-perf TESTSUITEFLAGS="--rebuild"`` to generate cached
>>> +   databases.
>>> +
>>> +.. note::
>>> +   This step may take some time depending on the number of tests that are being
>>> +   rebuilt, the complexity of the tests and the performance of the test machine.
>>> +   If you are only using one test, you can specify the test to run by adding the
>>> +   test number to the ``make`` command.
>>> +   (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``)
>>> +
>>> +2. Run ``make check-perf`` to measure the performance metric that you are
>>> +   benchmarking against. If you are only using one test, you can specify the test to run by adding the test number to the ``make`` command.
>>> +   (e.g. ``make check-perf TESTSUITEFLAGS="--rebuild <test number>"``)
>>
>> It's not very clear where the user would find the test results:
>>
>> tests/perf-testsuite.dir/results
> 
> It is mentioned further below but I will move it up to here to make it
> clearer.

Ah, OK, I missed it completely.

[...]

>>
>> The file is called perf-northd.at so if we ever decide to add some tests
>> to measure end-to-end performance including ovn-controller I assume
>> we'll need a new file, e.g., perf-controller.at.  Shouldn't we already
>> move all of the generic macros above to a shared file; maybe ovn-macros.at?
>>
> 
> This is exactly what I was thinking but I didn't want to do it yet
> because a) I am not sure if it will be used much b) I am unsure what
> will be common yet.
> 
> As such, I would prefer to hold off but I am happy to do it now if you
> think there is benefit.
> 

Ok, makes sense, let's keep it like this and see where it goes.

Thanks!



More information about the dev mailing list