[ovs-dev] [PATCH] CONTRIBUTING: Strengthen testing guidelines

[Russell Bryant]

On 02/08/2016 01:03 PM, Justin Pettit wrote:
> 
>> On Feb 8, 2016, at 9:34 AM, Kavanagh, Mark B <mark.b.kavanagh at intel.com> wrote:
>>
>>> I think the example that you show is actually a pretty good example of
>>> how quickly we catch problems.
>>
>> Sure. At the same time though, this is a reactive approach, rather than a proactive one, which inevitably results in more work for everyone (think fixing broken builds, failed CI builds, root-causing issues, etc, etc.). 
>>
>> If the validation can't be performed by patch authors, would you perhaps consider setting up an automated build environment on Nicera's end? This could be as simple as a git hook that would automatically build and test patches that have been approved by maintainers, when they are pushed to the upstream repo. If a test or build fails, the patch is rejected. It may be a little bit of work upfront, but could reduce the effort involved in fixing 'erroneous' patches, ultimately saving time for everyone.
> 
> We don't have the infrastructure to host that, but I don't think it
> would be difficult to get a third party to handle it.  If it can't be
> made to work someplace like Travis, maybe Intel could host it.  :-)

travis-ci is already doing a dpdk build at least, right?  So, build
errors are already quickly caught by CI (visibility of those failures is
another question).

So the issue is running the test suite?  I'd love to see a way to run
the test suite with OVS+DPDK that can be run in a virtual test
environment.  That'd remove the barriers from devs running it and also
let us run it in travis-ci.

> I would suggest that an email be generated and sent to the author and
> the mailing list rather than revert the change, which could be
> difficult if further patches were applied after the commit but before
> the build-check completed.  

There are solutions to this problem, but it requires letting a CI system
do the merging of patches for you.

> The Linux kernel does something similar.
> Shame is often a good motivator.

I don't think shame is exactly the culture I'd like to promote.
However, having travis-ci email committers whenever CI fails on a patch
they pushed would be helpful.  It's easy to miss otherwise.

Maybe that's possible and I'm just missing the option in travis-ci ...

-- 
Russell Bryant