[ovs-dev] [RFC] Federating the 0-day robot, and improving the testing
Aaron Conole
aconole at bytheb.org
Tue Dec 4 20:08:09 UTC 2018
Ben Pfaff <blp at ovn.org> writes:
> On Thu, Sep 06, 2018 at 04:56:18AM -0400, Aaron Conole wrote:
>> As of June, the 0-day robot has tested over 450 patch series.
>> Occasionally it spams the list (apologies for that), but for the
>> majority of the time it has caught issues before they made it to the
>> tree - so it's accomplishing the initial goal just fine.
>>
>> I see lots of ways it can improve. Currently, the bot runs on a light
>> system. It takes ~20 minutes to complete a set of tests, including all
>> the checkpatch and rebuild runs. That's not a big issue. BUT, it does
>> mean that the machine isn't able to perform all the kinds of regression
>> tests that we would want. I want to improve this in a way that various
>> contributors can bring their own hardware and regression tests to the
>> party. In that way, various projects can detect potential issues before
>> they would ever land on the tree and it could flag functional changes
>> earlier in the process.
>>
>> I'm not sure the best way to do that. One thing I'll be doing is
>> updating the bot to push a series that successfully builds and passes
>> checkpatch to a special branch on a github repository to kick off travis
>> builds. That will give us a more complete regression coverage, and we
>> could be confident that a series won't break something major. After
>> that, I'm not sure how to notify various alternate test infrastructures
>> how to kick off their own tests using the patched sources.
>
> That's pretty exciting.
>
> Don't forget about appveyor, either. Hardly any of us builds on
> Windows, so appveyor is likely to catch things that we won't.
I haven't forgotten about this.
Unveiling (after some quick testing... hopefully it continues to work):
https://github.com/ovsrobot/ovs
https://travis-ci.org/ovsrobot/ovs
https://ci.appveyor.com/project/ovsrobot/ovs
I don't know if the appveyor stuff is set up correctly, so if anyone who
is more knowledgeable in that area wants to assist contact me.
I only have one patch from one series built. BUT - every series now is
submitted to a github branch, and that kicks off Travis (and hopefully
appveyor). They can be found by their series id in patchwork:
series_SERIES_ID
I don't have any mechanism to clean them up. Presumably I'll need to
figure out how to do that at some point... :)
>> My goal is to get really early feedback on patch series. I've sent this
>> out to the folks I know are involved in testing and test discussions in
>> the hopes that we can talk about how best to get more CI happening. The
>> open questions:
>>
>> 1. How can we notify various downstream consumers of OvS of these
>> 0-day builds? Should we just rely on people rolling their own?
>> Should there be a more formalized framework? How will these other
>> test frameworks report any kind of failures?
>
> Do you mean notify of successes or failures? I assumed that the robot's
> email would notify us of that.
>
> Do you mean actually provide the builds? I don't know a good way to do
> that.
>
>> 2. What kinds of additional testing do we want to see the robot include?
>> Should the test results be made available in general on some kind of
>> public facing site? Should it just stay as a "bleep bloop -
>> failure!" marker?
>
> It would be super awesome if we could run the various additional
> testsuites that we have: check-system-userspace, check-kernel, etc. We
> can't run them easily on travis because they require superuser (and bugs
> sometimes crash the system).
>
>> 3. What other concerns should be addressed?
More information about the dev
mailing list