[ovs-dev] Reproducing ovn-scale-test results

[Valentine Sinitsyn]

Hi,

On 17.03.2017 02:24, Han Zhou wrote:
>
>
> On Thu, Mar 16, 2017 at 1:06 PM, Valentine Sinitsyn
> <valentine.sinitsyn at gmail.com <mailto:valentine.sinitsyn at gmail.com>> wrote:
>>
>> Hi Han,
>>
>> Thanks for the quick answer.
>>
>> On 17.03.2017 00:34, Han Zhou wrote:
>>>
>>> On Thu, Mar 16, 2017 at 3:58 AM, Valentine Sinitsyn
>>> <valentine.sinitsyn at gmail.com <mailto:valentine.sinitsyn at gmail.com>
> <mailto:valentine.sinitsyn at gmail.com
> <mailto:valentine.sinitsyn at gmail.com>>> wrote:
>>>>
>>>>
>>>> Hi all,
>>>>
>>>> We are doing some stress testing on OVN 2.7, and wanted to reproduce
>>>
>>> results from the talk [1]. Looking at ovn-scale-test sources, I have two
>>> questions:
>>>>
>>>>
>>>
>>> Hi Valentine,
>>>
>>> Thanks for picking this up.
>>>
>>>
>>>> - Do I get correctly that the benchmark always starts with the empty
>>>
>>> northbound db. Then lswitches are added, then you add ports to each
> lswitch?
>>>>
>>>>
>>>
>>> Yes, the test result shown in the talk was started from empty to
>>> gradually reach 20k lports on 200 lswitches.
>>>
>>>> - What is the batch size in port_create_args?
>>>
>>>
>>> I remember it was 100. In addition, there were 5 jobs running in
> parallel.
>>> +Lei to confirm.
>>
>> Could you recall how long (approximately) does it take to create and
> bind 20K ports with these settings? This would be really helpful.
>>
>
> I don't have the raw data now, but it took around 1 - 2 hours.
> We don't always run the full test, but after the full test is completed,
> we can just run another task to create and bind 1k more lports to
> evaluate the optimizations in each iteration on top of the existing scale.
Did you restart controllers or ovn-northd after running full tests 
before binding more ports. It would be interesting to learn how long 
does it take warm up IDL in controllers/northd in your setup.

>
> One more thing, the graph shared also involved sandbox (simulated HV)
> creation and lswitch creation. They were all created gradually during
> the test run.
> The flow was like:
> 1. create 50 sandboxes
> 2. (5 jobs in parallel) create 1 lswitch, create 100 lports, bind 100
> lports, wait 100 lport up
> 3. if there are 100 sandboxes already on the BM, switch to another BM
> 4. goto step1, until it is done for all 20 BMs.
>
> Moreover, in current ovs-scale-test code, the step "wait 100 lport up"
> is updated utilizing a new feature (wait for HVs to catch up) that was
Are you referring to ovn-nbctl --wait-until, or something else? If you 
were not using it for the talk, what exactly does "create + bind" mean 
on the graph?

Many thanks again.

Valentine

> added after the report, and we didn't run the test again yet with this
> change. I would expect it impact the test result slightly negatively,
> but it would more accurate.
>
>>>
>>>>
>>>> In short: is it true that for the setup involving (say) 10000 ports
>>>
>>> spanned over 100 lswitches in the aforementioned test, a Rally task
>>> would look like this?
>>>>
>>>>
>>>> {
>>>>     "version": 2,
>>>>     "title": "Create and bind port",
>>>>     "subtasks": [{
>>>>         "title": "Create and bind port",
>>>>         "workloads": [{
>>>>             "name": "OvnNetwork.create_and_bind_ports",
>>>>             "args": {
>>>>                 "network_create_args": {
>>>>                     "amount": 100,
>>>>                     "batch": 1,
>>>>                     "start_cidr": "172.16.1.0/24
> <http://172.16.1.0/24> <http://172.16.1.0/24>",
>>>>                     "physical_network": "providernet"
>>>>                 },
>>>>                 "port_create_args" : {"batch": 2},
>>>>                 "ports_per_network": 100,
>>>>                 "port_bind_args": {"wait_up": true}
>>>>             },
>>>>             "runner": {
>>>>                 "type": "serial","times": 1},
>>>>             "context": {
>>>>                "ovn_multihost" : {
>>>>                     "controller": "ovn-controller-node"
>>>>                 },
>>>>                 "sandbox":{ "tag": "ToR1"}
>>>>             }
>>>>         }]
>>>>     }]
>>>> }
>>>>
>>>> 1. https://youtu.be/okralc7LrZo?t=1185
>>>>
>>>> Thanks,
>>>> Valentine
>>>
>>>
>>
>> --
>> С уважением,
>> Синицын Валентин
>