[ovs-dev] [RFC] Federating the 0-day robot, and improving the testing

Aaron Conole aconole at bytheb.org
Tue Dec 4 20:08:09 UTC 2018


Ben Pfaff <blp at ovn.org> writes:

> On Thu, Sep 06, 2018 at 04:56:18AM -0400, Aaron Conole wrote:
>> As of June, the 0-day robot has tested over 450 patch series.
>> Occasionally it spams the list (apologies for that), but for the
>> majority of the time it has caught issues before they made it to the
>> tree - so it's accomplishing the initial goal just fine.
>> 
>> I see lots of ways it can improve.  Currently, the bot runs on a light
>> system.  It takes ~20 minutes to complete a set of tests, including all
>> the checkpatch and rebuild runs.  That's not a big issue.  BUT, it does
>> mean that the machine isn't able to perform all the kinds of regression
>> tests that we would want.  I want to improve this in a way that various
>> contributors can bring their own hardware and regression tests to the
>> party.  In that way, various projects can detect potential issues before
>> they would ever land on the tree and it could flag functional changes
>> earlier in the process.
>> 
>> I'm not sure the best way to do that.  One thing I'll be doing is
>> updating the bot to push a series that successfully builds and passes
>> checkpatch to a special branch on a github repository to kick off travis
>> builds.  That will give us a more complete regression coverage, and we
>> could be confident that a series won't break something major.  After
>> that, I'm not sure how to notify various alternate test infrastructures
>> how to kick off their own tests using the patched sources.
>
> That's pretty exciting.
>
> Don't forget about appveyor, either.  Hardly any of us builds on
> Windows, so appveyor is likely to catch things that we won't.

I haven't forgotten about this.

Unveiling (after some quick testing... hopefully it continues to work):

   https://github.com/ovsrobot/ovs
   https://travis-ci.org/ovsrobot/ovs
   https://ci.appveyor.com/project/ovsrobot/ovs

I don't know if the appveyor stuff is set up correctly, so if anyone who
is more knowledgeable in that area wants to assist contact me.

I only have one patch from one series built.  BUT - every series now is
submitted to a github branch, and that kicks off Travis (and hopefully
appveyor).  They can be found by their series id in patchwork:

   series_SERIES_ID

I don't have any mechanism to clean them up.  Presumably I'll need to
figure out how to do that at some point... :)

>> My goal is to get really early feedback on patch series.  I've sent this
>> out to the folks I know are involved in testing and test discussions in
>> the hopes that we can talk about how best to get more CI happening.  The
>> open questions:
>> 
>> 1. How can we notify various downstream consumers of OvS of these
>>    0-day builds?  Should we just rely on people rolling their own?
>>    Should there be a more formalized framework?  How will these other
>>    test frameworks report any kind of failures?
>
> Do you mean notify of successes or failures?  I assumed that the robot's
> email would notify us of that.
>
> Do you mean actually provide the builds?  I don't know a good way to do
> that.
>
>> 2. What kinds of additional testing do we want to see the robot include?
>>    Should the test results be made available in general on some kind of
>>    public facing site?  Should it just stay as a "bleep bloop -
>>    failure!" marker?
>
> It would be super awesome if we could run the various additional
> testsuites that we have: check-system-userspace, check-kernel, etc.  We
> can't run them easily on travis because they require superuser (and bugs
> sometimes crash the system).
>
>> 3. What other concerns should be addressed?


More information about the dev mailing list