[ovs-dev] [RFC] Federating the 0-day robot, and improving the testing

Ben Pfaff blp at ovn.org
Fri Sep 7 17:24:56 UTC 2018


On Thu, Sep 06, 2018 at 04:56:18AM -0400, Aaron Conole wrote:
> As of June, the 0-day robot has tested over 450 patch series.
> Occasionally it spams the list (apologies for that), but for the
> majority of the time it has caught issues before they made it to the
> tree - so it's accomplishing the initial goal just fine.
> 
> I see lots of ways it can improve.  Currently, the bot runs on a light
> system.  It takes ~20 minutes to complete a set of tests, including all
> the checkpatch and rebuild runs.  That's not a big issue.  BUT, it does
> mean that the machine isn't able to perform all the kinds of regression
> tests that we would want.  I want to improve this in a way that various
> contributors can bring their own hardware and regression tests to the
> party.  In that way, various projects can detect potential issues before
> they would ever land on the tree and it could flag functional changes
> earlier in the process.
> 
> I'm not sure the best way to do that.  One thing I'll be doing is
> updating the bot to push a series that successfully builds and passes
> checkpatch to a special branch on a github repository to kick off travis
> builds.  That will give us a more complete regression coverage, and we
> could be confident that a series won't break something major.  After
> that, I'm not sure how to notify various alternate test infrastructures
> how to kick off their own tests using the patched sources.

That's pretty exciting.

Don't forget about appveyor, either.  Hardly any of us builds on
Windows, so appveyor is likely to catch things that we won't.

> My goal is to get really early feedback on patch series.  I've sent this
> out to the folks I know are involved in testing and test discussions in
> the hopes that we can talk about how best to get more CI happening.  The
> open questions:
> 
> 1. How can we notify various downstream consumers of OvS of these
>    0-day builds?  Should we just rely on people rolling their own?
>    Should there be a more formalized framework?  How will these other
>    test frameworks report any kind of failures?

Do you mean notify of successes or failures?  I assumed that the robot's
email would notify us of that.

Do you mean actually provide the builds?  I don't know a good way to do
that.

> 2. What kinds of additional testing do we want to see the robot include?
>    Should the test results be made available in general on some kind of
>    public facing site?  Should it just stay as a "bleep bloop -
>    failure!" marker?

It would be super awesome if we could run the various additional
testsuites that we have: check-system-userspace, check-kernel, etc.  We
can't run them easily on travis because they require superuser (and bugs
sometimes crash the system).

> 3. What other concerns should be addressed?


More information about the dev mailing list