[ovs-dev] [PATCH] Testing: Introduce "bring your own lab" as an additional test option
Stokes, Ian
ian.stokes at intel.com
Fri Jul 16 16:05:15 UTC 2021
> Two big changes are announced here, and the first pertains to
> allowing other developers and labs to report patch status in the
> ozlabs patchwork UI to assist maintainers making decisions on
> patch inclusion.
>
> To make this as easy as possible, the robot will take a known form
> of email sent to the ovs-build at openvswitch.org mailing list and
> push the information into patchwork. This information can be browsed
> either via the web UI or programmatically via the checks API.
>
> The second change is to de-emphasize the use of Travis-CI as a project
> wide resource. Travis-CI might still be useful for individual developers
> testing small patches, but can no longer fulfill the role of project-wide
> continuous integration service. Remove the badge from the front page, and
> note the other CI services that OVS supports. Rather than completely drop
> all references to travis, simply emphasize using GitHub Actions as the
> preferred CI solution.
>
Hi Aaron, thanks for driving this. A few comments below but overall agree with this process.
> Signed-off-by: Aaron Conole <aconole at redhat.com>
> Refs: https://mail.openvswitch.org/pipermail/ovs-dev/2021-June/384040.html
> ---
> .../contributing/submitting-patches.rst | 8 +-
> Documentation/topics/testing.rst | 122 ++++++++++++------
> NEWS | 3 +
> README.rst | 2 -
> 4 files changed, 90 insertions(+), 45 deletions(-)
>
> diff --git a/Documentation/internals/contributing/submitting-patches.rst
> b/Documentation/internals/contributing/submitting-patches.rst
> index 4a6780371d..a6a10f8d89 100644
> --- a/Documentation/internals/contributing/submitting-patches.rst
> +++ b/Documentation/internals/contributing/submitting-patches.rst
> @@ -68,10 +68,10 @@ Testing is also important:
> feature. A bug fix patch should preferably add a test that would
> fail if the bug recurs.
>
> -If you are using GitHub, then you may utilize the travis-ci.org and the GitHub
> -Actions CI build systems. They will run some of the above tests automatically
> -when you push changes to your repository. See the "Continuous Integration
> with
> -Travis-CI" in :doc:`/topics/testing` for details on how to set it up.
> +If you are using GitHub, then you may utilize the the GitHub Actions CI build
> +systems. They will run some of the above tests automatically when you push
> +changes to your repository. See the "Continuous Integration" section in
> +:doc:`/topics/testing` for details on continuous integration.
>
> Email Subject
> -------------
> diff --git a/Documentation/topics/testing.rst
> b/Documentation/topics/testing.rst
> index 951fe9e851..7e0ced8855 100644
> --- a/Documentation/topics/testing.rst
> +++ b/Documentation/topics/testing.rst
> @@ -416,45 +416,89 @@ You should invoke scan-view to view analysis results.
> The last line of output
> from ``clang-analyze`` will list the command (containing results directory)
> that you should invoke to view the results on a browser.
>
> -Continuous Integration with Travis CI
> --------------------------------------
> -
> -A .travis.yml file is provided to automatically build Open vSwitch with various
> -build configurations and run the testsuite using Travis CI. Builds will be
> -performed with gcc, sparse and clang with the -Werror compiler flag included,
> -therefore the build will fail if a new warning has been introduced.
> -
> -The CI build is triggered via git push (regardless of the specific branch) or
> -pull request against any Open vSwitch GitHub repository that is linked to
> -travis-ci.
> -
> -Instructions to setup travis-ci for your GitHub repository:
> -
> -1. Go to https://travis-ci.org/ and sign in using your GitHub ID.
> -2. Go to the "Repositories" tab and enable the ovs repository. You may disable
> - builds for pushes or pull requests.
> -3. In order to avoid forks sending build failures to the upstream mailing list,
> - the notification email recipient is encrypted. If you want to receive email
> - notification for build failures, replace the encrypted string:
> -
> - 1. Install the travis-ci CLI (Requires ruby >=2.0): gem install travis
> - 2. In your Open vSwitch repository: travis encrypt mylist at mydomain.org
> - 3. Add/replace the notifications section in .travis.yml and fill in the
> - secure string as returned by travis encrypt::
> -
> - notifications:
> - email:
> - recipients:
> - - secure: "....."
> -
> - .. note::
> - You may remove/omit the notifications section to fall back to default
> - notification behaviour which is to send an email directly to the author and
> - committer of the failing commit. Note that the email is only sent if the
> - author/committer have commit rights for the particular GitHub repository.
> -
> -4. Pushing a commit to the repository which breaks the build or the
> - testsuite will now trigger a email sent to mylist at mydomain.org
> +Continuous Integration
> +----------------------
> +
> +The Open vSwitch project can make use of multiple public CI services to help
> +developers ensure their patches don't introduce additional regressions. Each
> +service requires different forms of configuration, and for the supported
> +services the configuration file(s) to automatically build Open vSwitch with
> +various build configurations and run the testsuites is/are available. For
> +example, the GitHub Actions builds will be performed with gcc, sparse and
> clang
> +with the -Werror compiler flag included, therefore the build will fail if a new
> +warning has been introduced.
> +
> +Each ci system uses a different method of enablement, but most are available
> +from the GitHub settings page. By default, Open vSwitch includes a GitHub
> +actions running configuration, and this will automatically email the repository
> +owner.
> +
> +Currently, Open vSwitch project enables the following public CI services along
> +with the appropriate configuration settings::
> +
> + - AppVeyor: appveyor.yml
> + - Cirrus-CI: .cirrus.yml
> + - GitHub Actions: .github/workflows
> + - Travis-CI: .travis.yml
Is it worth mentioning 'Read the Docs' for documentation builds?
> +
> +GitHub Actions is available without any additional configuration.
> +
> +Additionally, as each patch is posted to the mailing list, the public CI
> +machinery will run additional tests on the patches and report results.
> +
> +Bring Your Own Lab
> +------------------
> +
> +The Open vSwitch project makes use of the ozlabs patchwork instance for
> +patch status and management. This patch tracking tool provides information
> +on the state of changes proposed for Open vSwitch. One feature that the
> +patchwork instance offers is for the concept of 'Checks' to be reported.
> +These are sets of test functionality that can be executed and present
> +status 'at-a-glance' with the ability to drill deeper into why a particular
> +test failed.
> +
> +To this end, the Open vSwitch "0-day Robot" will accept emails sent to
> +ovs-build at openvswitch.org formatted in the right way to be reflected on this
> +page. This allows any lab to contribute to the testing and validation of
> +patches.
> +
> +To report a test status to a particular patch, send exactly one email to
> +the mailing formatted as such::
> +
> + From: your email <address at server.com>
> + To: ovs-build at openvswitch.org
> + Date: Mon, 28 Jun 2021 00:00:00 +0000
> + Subject: |STATUS| pwPATCHID commit subject
> +
> + Test-Label: your-robot-or-test-name
> + Test-Status: STATUS
> + http://patchwork.ozlabs.org/api/patches/PATCHID/
> +
> + _ONE LINE DESCRIPTION_
+1 really like the approach above. Looking to set something up in the Intel labs to submit to this ourselves.
> +
> + Addtional details
Is the purpose of additional details to provide info on the Hardware setup, OVS configuration, logs for tests that fail etc? Or do you envision that going elsewhere?
> +
> +In the above example, the STATUS is one of "success" "warning" "fail"
> depending
> +on the outcome of the test. This value is case-sensitive. PATCHID should be
> the
> +patch for which the test was executed, and ONE LINE DESCRIPTION should be
> a simple
> +one line result description, ex: "robot-test: success"
> +
> +It is strongly recommended that if the patch is reporting a failure case, one
> +of the patch authors is cc'd as well.
> +
> +Example::
> +
> + From: 0-Day Robot <robot at bytheb.org>
> + To: ovs-build at openvswitch.org
> + Date: Fri, 25 Jun 2021 14:30:37 +0400
> + Subject: |success| pw1497375 [ovs-dev] [PATCH v2] checkpatch: Ignore
> macro definitions of FOR_EACH
> +
> + Test-Label: github-robot
> + Test-Status: success
> + http://patchwork.ozlabs.org/api/patches/1497375/
> +
> + _github build: passed_
> + Build URL: https://github.com/ovsrobot/ovs/actions/runs/972188336
>
So this will work for functional testing, I'm wondering if we were to do regression testing for performance how we would approach it?
For sure if you failed due to performance there is probably a lot of detail required to understand in order to reproduce.
Maybe we're getting ahead of ourselves here and functional testing is a good place to start?
Regards
Ian
> vsperf
> ------
> diff --git a/NEWS b/NEWS
> index b9cb74e9e6..a771eec1a1 100644
> --- a/NEWS
> +++ b/NEWS
> @@ -21,6 +21,9 @@ Post-v2.15.0
> - ovsdb-tool:
> * New option '--election-timer' to the 'create-cluster' command to set the
> leader election timer during cluster creation.
> + - Testing:
> + * Integrate "Bring your own lab" to the build reports page, allowing any
> + testing lab to submit a test report for a patch.
>
>
> v2.15.0 - 15 Feb 2021
> diff --git a/README.rst b/README.rst
> index 319f705154..8e64d74aee 100644
> --- a/README.rst
> +++ b/README.rst
> @@ -8,8 +8,6 @@ Open vSwitch
>
> .. image::
> https://github.com/openvswitch/ovs/workflows/Build%20and%20Test/badge.s
> vg
> :target: https://github.com/openvswitch/ovs/actions
> -.. image:: https://travis-ci.org/openvswitch/ovs.png
> - :target: https://travis-ci.org/openvswitch/ovs
> .. image::
> https://ci.appveyor.com/api/projects/status/github/openvswitch/ovs?branch=
> master&svg=true&retina=true
> :target: https://ci.appveyor.com/project/blp/ovs/history
> .. image:: https://api.cirrus-ci.com/github/openvswitch/ovs.svg
> --
> 2.31.1
>
> _______________________________________________
> dev mailing list
> dev at openvswitch.org
> https://mail.openvswitch.org/mailman/listinfo/ovs-dev
More information about the dev
mailing list