[ovs-dev] Improved OVN CI

Mark Michelson mmichels at redhat.com
Fri Mar 27 18:24:31 UTC 2020

Hi everyone,

I've taken some time recently to look into the current CI for OVS/OVN, 
and I think there is room for improvement. Here I will outline a 4 part 
plan for improving CI for OVN.

Part 1: Report travis build failures as replies to patch submissions.

Currently, 0-day robot will respond to patches that fail to apply or 
fail to pass checkpatch.py. However, builds are handled by travis and 
reported out-of-band to the ovs-build list. By having build failures 
reported in response to the patch, it makes it much easier to correlate 
results to a patch series and does not require signing up for a separate 
mailing list.

I suspect this will lead to some noise at first as some tests may need 
to be stabilized. If we are diligent about fixing such volatile tests, 
then this should reduce the noise after a while. Once we reach a point 
where the signal-to-noise ratio is acceptable, we can move on to...

Part 2: Expand testing done by travis.

Travis runs `make distcheck` now. This could be expanded to run the 
* system tests
* ovn-kubernetes tests
* Run tests against all supported versions of OVS
CI could also be expanded to run some scale tests, but that likely would 
need to be done outside of travis.

Like with part 1, this likely will be noisy as more tests are added. 
Again, though, as we fix these problems, it will lead to a more stable 
suite of tests. We can then move on to...

Part 3: Correlate tests with patchwork checks

Patchwork has "checks" that can be set on submissions which correspond 
to external tests. The tests we add in parts 1 and 2 can correspond to 
patchwork checks. The checks can provide an easily-referenced way to 
determine the status of tests for a given patch series. Adding checks to 
patchwork also lays the groundwork for Part 4.

(Note: It may make sense to switch parts 2 and 3. We may want to 
establish patchwork checks prior to adding more tests)

Part 4: Gating

If we have a stable and extensive testsuite, then we can attempt to 
enforce gating on patch submissions. In other words, patches that do not 
pass all checks cannot be pushed. This can be enforced in a few ways

* Convention: We regulate ourselves. We agree as a community to only 
push changes if tests have passed, and we refuse to push patches that 
have failed CI.
* Automation: The bot that monitors patchwork for new submissions and 
starts CI could potentially merge patches that have received ACKs and 
have passed CI. Humans would not push patches themselves.
* Push hook: When pushing patches, if patchwork CI checks have not 
passed (and if the patch has not been ACKed), then the hook would reject 
the push.
* Pull Requests: Switching from mailing list patches to pull requests 
would allow for github to enforce gating.

In any of the above cases, there will be ways for committers to override 
the system if absolutely necessary (e.g. CI false positives)

What are your thoughts on this expansion of CI for OVN? Do you have 
ideas for other areas where it can be expanded? What are your thoughts 
on gating in particular?

Thanks for your feedback,
Mark Michelson

More information about the dev mailing list