[ovs-dev] [PATCH] github: Fix handling of python packages.

Ilya Maximets i.maximets at ovn.org
Mon Mar 15 12:04:41 UTC 2021


On 3/15/21 12:53 PM, Aaron Conole wrote:
> Ilya Maximets <i.maximets at ovn.org> writes:
> 
>> GitHub Actions doesn't have python locations in PATH and different
>> runners might have different configuration for default python
>> location and versions.  For example, on some runners python2 might
>> be installed or not.
>>
>> Missing PATH causes weird situations on older branches where during
>> one run our scripts can locate just installed flake8 and can't do
>> that on a different run.  But this might also create other
>> unpredictable issues on all branches.
> 
> This is strange.  We do an install via the package manager - is it not
> installing to the correct PATH location?

We are installing 'flake8' and 'hacking' by pip and these are installed
to python locations that are not in PATH.

> 
>> It's required to use actions/setup-python at v2 in order to have
>> predictable version of python installed and paths correctly configured.
>> Due to some bugs in GHA itself it doesn't set $HOME/.local/bin into
>> PATH, so we have to do that manually for now in order to use '--user'.
>> This might be fixed later in actions/setup-python or in base runners.
>> We already setting it for DPDK 20.11 (I think the issue was spotted
>> but not fully investigated).  Moving PATH updates to a separate step
>> to make them more explicit and avaialble for all steps of the job.
> 
> *available
> 
> Do you know if GHA team has a plan to fix this?

There are 2 issues:
1. For general virtual-environments:
     https://github.com/actions/virtual-environments/issues/2455
   They tried and failed. :)
   Promised to fix once they figure out what happened, but I'd not trust them.

2. For setup-python action:
     https://github.com/actions/setup-python/issues/99
   State is unclear to me.

> 
> It's a strange syntax, but it looks like the manual requires that we do
> PATH updates via 'echo "..." >> $GITHUB_PATH' so, LGTM.

Yeah, never used this syntax before.

> 
>> Unfortunately actions/setup-python at v2 also makes invisible python
>> packages installed from Ubuntu repositories.  Switching them to
>> 'pip3 install'.
>>
>> Fixes: 6cb2f5a630e3 ("github: Add GitHub Actions workflow.")
>> Reported-by: Numan Siddique <numans at ovn.org>
>> Signed-off-by: Ilya Maximets <i.maximets at ovn.org>
>> ---
>>  .ci/linux-prepare.sh                 |  3 ++-
>>  .github/workflows/build-and-test.yml | 26 +++++++++++++++++++++-----
>>  2 files changed, 23 insertions(+), 6 deletions(-)
>>
>> diff --git a/.ci/linux-prepare.sh b/.ci/linux-prepare.sh
>> index 69a40011f..c55125cf7 100755
>> --- a/.ci/linux-prepare.sh
>> +++ b/.ci/linux-prepare.sh
>> @@ -20,7 +20,8 @@ cd sparse
>>  make -j4 HAVE_LLVM= HAVE_SQLITE= install
>>  cd ..
>>  
>> -pip3 install --disable-pip-version-check --user flake8 hacking
>> +pip3 install --disable-pip-version-check --user \
>> +    flake8 hacking sphinx pyOpenSSL wheel setuptools
>>  pip3 install --user --upgrade docutils
>>  pip3 install --user  'meson==0.47.1'
>>  
>> diff --git a/.github/workflows/build-and-test.yml b/.github/workflows/build-and-test.yml
>> index 1bb72bbb1..ce98a9f98 100644
>> --- a/.github/workflows/build-and-test.yml
>> +++ b/.github/workflows/build-and-test.yml
>> @@ -8,9 +8,7 @@ jobs:
>>        dependencies: |
>>          automake libtool gcc bc libjemalloc1 libjemalloc-dev    \
>>          libssl-dev llvm-dev libelf-dev libnuma-dev libpcap-dev  \
>> -        ninja-build python3-openssl python3-pip                 \
>> -        python3-setuptools python3-sphinx python3-wheel         \
>> -        selinux-policy-dev
>> +        ninja-build selinux-policy-dev
>>        deb_dependencies: |
>>          linux-headers-$(uname -r) build-essential fakeroot devscripts equivs
>>        AFXDP:       ${{ matrix.afxdp }}
>> @@ -115,6 +113,16 @@ jobs:
>>      - name: checkout
>>        uses: actions/checkout at v2
>>  
>> +    - name: update PATH
>> +      run:  |
>> +        echo "$HOME/bin"        >> $GITHUB_PATH
>> +        echo "$HOME/.local/bin" >> $GITHUB_PATH
>> +
>> +    - name: set up python
>> +      uses: actions/setup-python at v2
>> +      with:
>> +        python-version: '3.x'
>> +
>>      - name: create ci signature file for the dpdk cache key
>>        if:   matrix.dpdk != '' || matrix.dpdk_shared != ''
>>        # This will collect most of DPDK related lines, so hash will be different
>> @@ -151,7 +159,7 @@ jobs:
>>        run:  ./.ci/linux-prepare.sh
>>  
>>      - name: build
>> -      run:  PATH="$PATH:$HOME/bin:$HOME/.local/bin" ./.ci/linux-build.sh
>> +      run:  ./.ci/linux-build.sh
>>  
>>      - name: upload deb packages
>>        if:   matrix.deb_package != ''
>> @@ -194,12 +202,20 @@ jobs:
>>      steps:
>>      - name: checkout
>>        uses: actions/checkout at v2
>> +    - name: update PATH
>> +      run:  |
>> +        echo "$HOME/bin"        >> $GITHUB_PATH
>> +        echo "$HOME/.local/bin" >> $GITHUB_PATH
>> +    - name: set up python
>> +      uses: actions/setup-python at v2
>> +      with:
>> +        python-version: '3.x'
>>      - name: install dependencies
>>        run:  brew install automake libtool
>>      - name: prepare
>>        run:  ./.ci/osx-prepare.sh
>>      - name: build
>> -      run:  PATH="$PATH:$HOME/bin" ./.ci/osx-build.sh
>> +      run:  ./.ci/osx-build.sh
>>      - name: upload logs on failure
>>        if: failure()
>>        uses: actions/upload-artifact at v2
> 



More information about the dev mailing list