[ovs-dev] Improved OVN CI

Dumitru Ceara dceara at redhat.com
Thu Sep 17 18:51:10 UTC 2020

On 3/27/20 7:24 PM, Mark Michelson wrote:
> Hi everyone,
> I've taken some time recently to look into the current CI for OVS/OVN,
> and I think there is room for improvement. Here I will outline a 4 part
> plan for improving CI for OVN.
> Part 1: Report travis build failures as replies to patch submissions.
> Currently, 0-day robot will respond to patches that fail to apply or
> fail to pass checkpatch.py. However, builds are handled by travis and
> reported out-of-band to the ovs-build list. By having build failures
> reported in response to the patch, it makes it much easier to correlate
> results to a patch series and does not require signing up for a separate
> mailing list.
> I suspect this will lead to some noise at first as some tests may need
> to be stabilized. If we are diligent about fixing such volatile tests,
> then this should reduce the noise after a while. Once we reach a point
> where the signal-to-noise ratio is acceptable, we can move on to...
> Part 2: Expand testing done by travis.
> Travis runs `make distcheck` now. This could be expanded to run the
> following:
> * system tests
> * ovn-kubernetes tests
> * Run tests against all supported versions of OVS
> CI could also be expanded to run some scale tests, but that likely would
> need to be done outside of travis.
> Like with part 1, this likely will be noisy as more tests are added.
> Again, though, as we fix these problems, it will lead to a more stable
> suite of tests. We can then move on to...

Bumping this thread. As mentioned during the IRC meeting today [1], I
think it might be useful to spend some time in this direction.

One of the suggestions during the meeting was to move to github actions
so that we reuse the ovn-k8s CI infrastructure.

There was also the question whether there's any missing feature in
TravisCI that wouldn't allow us to run the same ovn-k8s tests in Travis.
One advantage of this is, I guess, the fact that the ovsrobot already
triggers Travis runs for every patch submission (result emails go to
ovs-build ML).

Another direction, orthogonal I would say, could be to harden the
existing OVN tests, ovn*.at. A potential first step could be to
investigate the coverage results (make check-lcov) and see if there are
areas where we need more tests.

Looking forward to hearing more thoughts on this matter :)



> Part 3: Correlate tests with patchwork checks
> Patchwork has "checks" that can be set on submissions which correspond
> to external tests. The tests we add in parts 1 and 2 can correspond to
> patchwork checks. The checks can provide an easily-referenced way to
> determine the status of tests for a given patch series. Adding checks to
> patchwork also lays the groundwork for Part 4.
> (Note: It may make sense to switch parts 2 and 3. We may want to
> establish patchwork checks prior to adding more tests)
> Part 4: Gating
> If we have a stable and extensive testsuite, then we can attempt to
> enforce gating on patch submissions. In other words, patches that do not
> pass all checks cannot be pushed. This can be enforced in a few ways
> * Convention: We regulate ourselves. We agree as a community to only
> push changes if tests have passed, and we refuse to push patches that
> have failed CI.
> * Automation: The bot that monitors patchwork for new submissions and
> starts CI could potentially merge patches that have received ACKs and
> have passed CI. Humans would not push patches themselves.
> * Push hook: When pushing patches, if patchwork CI checks have not
> passed (and if the patch has not been ACKed), then the hook would reject
> the push.
> * Pull Requests: Switching from mailing list patches to pull requests
> would allow for github to enforce gating.
> In any of the above cases, there will be ways for committers to override
> the system if absolutely necessary (e.g. CI false positives)
> What are your thoughts on this expansion of CI for OVN? Do you have
> ideas for other areas where it can be expanded? What are your thoughts
> on gating in particular?
> Thanks for your feedback,
> Mark Michelson

More information about the dev mailing list