[ovs-dev] [RFC] Federating the 0-day robot, and improving the testing

[Eelco Chaudron]

On 11 Sep 2018, at 17:51, Aaron Conole wrote:

> "Eelco Chaudron" <echaudro at redhat.com> writes:
>
>> On 6 Sep 2018, at 10:56, Aaron Conole wrote:
>>
>>> As of June, the 0-day robot has tested over 450 patch series.
>>> Occasionally it spams the list (apologies for that), but for the
>>> majority of the time it has caught issues before they made it to the
>>> tree - so it's accomplishing the initial goal just fine.
>>>
>>> I see lots of ways it can improve.  Currently, the bot runs on a 
>>> light
>>> system.  It takes ~20 minutes to complete a set of tests, including
>>> all
>>> the checkpatch and rebuild runs.  That's not a big issue.  BUT, it
>>> does
>>> mean that the machine isn't able to perform all the kinds of
>>> regression
>>> tests that we would want.  I want to improve this in a way that
>>> various
>>> contributors can bring their own hardware and regression tests to 
>>> the
>>> party.  In that way, various projects can detect potential issues
>>> before
>>> they would ever land on the tree and it could flag functional 
>>> changes
>>> earlier in the process.
>>>
>>> I'm not sure the best way to do that.  One thing I'll be doing is
>>> updating the bot to push a series that successfully builds and 
>>> passes
>>> checkpatch to a special branch on a github repository to kick off
>>> travis
>>> builds.  That will give us a more complete regression coverage, and 
>>> we
>>> could be confident that a series won't break something major.  After
>>> that, I'm not sure how to notify various alternate test
>>> infrastructures
>>> how to kick off their own tests using the patched sources.
>>>
>>> My goal is to get really early feedback on patch series.  I've sent
>>> this
>>> out to the folks I know are involved in testing and test discussions
>>> in
>>> the hopes that we can talk about how best to get more CI happening.
>>> The
>>> open questions:
>>>
>>> 1. How can we notify various downstream consumers of OvS of these
>>>    0-day builds?  Should we just rely on people rolling their own?
>>>    Should there be a more formalized framework?  How will these 
>>> other
>>>    test frameworks report any kind of failures?
>>>
>>> 2. What kinds of additional testing do we want to see the robot
>>> include?
>>
>> First of all thanks for the 0-day robot, I really like the idea…
>>
>> One thing I feel would really benefit is some basic performance
>> testing, like a PVP test for the kernel/dpdk datapath. This will help
>> easily identifying performance impacting patches as they happen…
>> Rather than people figuring out after a release why their performance
>> has dropped.
>
> Yes - I hope to pull in the work you've done for ovs_perf to have some
> kind of baselines.
>
> For this to make sense, I think it also needs to have a bunch of
> hardware that we can benchmark (hint hint to some of the folks in the 
> CC
> list :).  Not for absolute numbers, but at least to detect significant
> changes.
>
> I'm also not sure how to measure a 'problem.'  Do we run a test
> pre-series, and then run it post-series?  In that case, we could 
> slowly
> degrade performance over time without any noticing.  Do we take it 
> from
> the previous release, and compare?  Might make more sense, but I don't
> know if it has other problems associated.  What are the thresholds we
> use for saying something is a regression?  How do we report it to
> developers?

Guess both in an ideal world, and maybe add a weekly baseline for master 
:)

Having a graph of this would be really nice. However, this might be a 
whole project on itself, i.e. performance runs on all commits to 
master…


>>>    Should the test results be made available in general on some kind
>>> of
>>>    public facing site?  Should it just stay as a "bleep bloop -
>>>    failure!" marker?
>>>
>>> 3. What other concerns should be addressed?
>>> _______________________________________________
>>> dev mailing list
>>> dev at openvswitch.org
>>> https://mail.openvswitch.org/mailman/listinfo/ovs-dev