[ovs-dev] Improved OVN CI

Mark Gray mark.d.gray at redhat.com
Fri Sep 18 16:44:18 UTC 2020

On 17/09/2020 19:51, Dumitru Ceara wrote:
> On 3/27/20 7:24 PM, Mark Michelson wrote:
>> Hi everyone,
>> I've taken some time recently to look into the current CI for OVS/OVN,
>> and I think there is room for improvement. Here I will outline a 4 part
>> plan for improving CI for OVN.
>> Part 1: Report travis build failures as replies to patch submissions.
>> Currently, 0-day robot will respond to patches that fail to apply or
>> fail to pass checkpatch.py. However, builds are handled by travis and
>> reported out-of-band to the ovs-build list. By having build failures
>> reported in response to the patch, it makes it much easier to correlate
>> results to a patch series and does not require signing up for a separate
>> mailing list.
>> I suspect this will lead to some noise at first as some tests may need
>> to be stabilized. If we are diligent about fixing such volatile tests,
>> then this should reduce the noise after a while. Once we reach a point
>> where the signal-to-noise ratio is acceptable, we can move on to...

+1 - I find this useful on the OVS list
>> Part 2: Expand testing done by travis.
>> Travis runs `make distcheck` now. This could be expanded to run the
>> following:
>> * system tests
>> * ovn-kubernetes tests
>> * Run tests against all supported versions of OVS
>> CI could also be expanded to run some scale tests, but that likely would
>> need to be done outside of travis.

Any idea how long it would take to run all these? I would prefer that I
ran them before I submitted a patch but if they took too long, I may not
do that which would mean there would be more noise on the list.
>> Like with part 1, this likely will be noisy as more tests are added.
>> Again, though, as we fix these problems, it will lead to a more stable
>> suite of tests. We can then move on to...

What about coverage tests?

> Bumping this thread. As mentioned during the IRC meeting today [1], I
> think it might be useful to spend some time in this direction.
> One of the suggestions during the meeting was to move to github actions
> so that we reuse the ovn-k8s CI infrastructure.
> There was also the question whether there's any missing feature in
> TravisCI that wouldn't allow us to run the same ovn-k8s tests in Travis.
> One advantage of this is, I guess, the fact that the ovsrobot already
> triggers Travis runs for every patch submission (result emails go to
> ovs-build ML).
> Another direction, orthogonal I would say, could be to harden the
> existing OVN tests, ovn*.at. A potential first step could be to
> investigate the coverage results (make check-lcov) and see if there are
> areas where we need more tests.
> Looking forward to hearing more thoughts on this matter :)

What about badges on "README.md" to report failing/succeeding builds or
coverage or something like that.
> Thanks,
> Dmitru
> [1]
> http://eavesdrop.openstack.org/meetings/ovn_community_development_discussion/2020/ovn_community_development_discussion.2020-09-17-17.20.log.html
>> Part 3: Correlate tests with patchwork checks
>> Patchwork has "checks" that can be set on submissions which correspond
>> to external tests. The tests we add in parts 1 and 2 can correspond to
>> patchwork checks. The checks can provide an easily-referenced way to
>> determine the status of tests for a given patch series. Adding checks to
>> patchwork also lays the groundwork for Part 4.
>> (Note: It may make sense to switch parts 2 and 3. We may want to
>> establish patchwork checks prior to adding more tests)

Sounds good

>> Part 4: Gating
>> If we have a stable and extensive testsuite, then we can attempt to
>> enforce gating on patch submissions. In other words, patches that do not
>> pass all checks cannot be pushed. This can be enforced in a few ways
>> * Convention: We regulate ourselves. We agree as a community to only
>> push changes if tests have passed, and we refuse to push patches that
>> have failed CI.
>> * Automation: The bot that monitors patchwork for new submissions and
>> starts CI could potentially merge patches that have received ACKs and
>> have passed CI. Humans would not push patches themselves.

Sound a little dangerous - don't know if I want the robots taking over!

>> * Push hook: When pushing patches, if patchwork CI checks have not
>> passed (and if the patch has not been ACKed), then the hook would reject
>> the push.
>> * Pull Requests: Switching from mailing list patches to pull requests
>> would allow for github to enforce gating.
>> In any of the above cases, there will be ways for committers to override
>> the system if absolutely necessary (e.g. CI false positives)>>
>> What are your thoughts on this expansion of CI for OVN? Do you have
>> ideas for other areas where it can be expanded? What are your thoughts
>> on gating in particular?
>> Thanks for your feedback,

Thanks for this Mark

>> Mark Michelson
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev

More information about the dev mailing list