Can't run envs that aren't in envlist?

[AdamWill]

I'm currently trying to set this up for a repo, but I'm having a bit of trouble with one specific thing.

It seems this can't handle environments that aren't in envlist?

The idea here is that I want to have -ci environments which aren't used if you just run tox normally. They're only used when explicitly specified. That's why I don't want to put them in envlist: whatever you put in envlist is what will be run if you just run tox with no -e argument, and there's no way to change this, so far as I can tell.

The config I have:

[tox]
envlist = py{27,36,37,38,39}

[gh-actions]
python =
    2.7: py27
    3.6: py36-ci
    3.7: py37-ci
    3.8: py38-ci
    3.9: py39-ci

[testenv]
deps =
    -r{toxinidir}/install.requires
    -r{toxinidir}/tests.requires
    ci: -r{toxinidir}/ci.requires

commands =
    py.test
    ci: py.test --cov-report term-missing --cov-report xml --cov openqa_client
    ci: diff-cover coverage.xml --fail-under=90
    ci: diff-quality --violations=pylint --fail-under=90
setenv =
    PYTHONPATH = {toxinidir}

works fine when I test from the command line. If I just run tox, the py27, py36 etc. environments - which don't run the CI commands - are run. If I run tox -epy{27,36,37,38,39}-ci, tox runs the CI environments and commands.

But if you look at the test results in the pull request, this doesn't seem to be working there. In tox.ini I configured all the Python 3 environments as I actually want them to be set up:

3.6: py36-ci
3.7: py37-ci
3.8: py38-ci
3.9: py39-ci

but it seems like when tox-gh-actions goes to run these, it just sort of does nothing:

2020-02-28T01:10:13.0230470Z ##[group]Run tox
2020-02-28T01:10:13.0230678Z �[36;1mtox�[0m
2020-02-28T01:10:13.0267544Z shell: /bin/bash -e {0}
2020-02-28T01:10:13.0267665Z env:
2020-02-28T01:10:13.0267776Z   pythonLocation: /opt/hostedtoolcache/Python/3.6.10/x64
2020-02-28T01:10:13.0267870Z ##[endgroup]
2020-02-28T01:10:13.2767035Z ___________________________________ summary ____________________________________
2020-02-28T01:10:13.2769413Z   congratulations :)

I set the Python 2.7 line to use the non-CI environment, as a check:

2.7: py27

and just that environment works correctly:

2020-02-28T01:10:13.9845791Z ##[group]Run tox
2020-02-28T01:10:13.9845964Z �[36;1mtox�[0m
2020-02-28T01:10:13.9877602Z shell: /bin/bash -e {0}
2020-02-28T01:10:13.9877667Z env:
2020-02-28T01:10:13.9877745Z   pythonLocation: /opt/hostedtoolcache/Python/2.7.17/x64
2020-02-28T01:10:13.9877826Z ##[endgroup]
2020-02-28T01:10:14.1993163Z py27 create: /home/runner/work/openQA-python-client/openQA-python-client/.tox/py27
2020-02-28T01:10:14.6157194Z py27 installdeps: -r/home/runner/work/openQA-python-client/openQA-python-client/install.requires, -r/home/runner/work/openQA-python-client/openQA-python-client/tests.requires
2020-02-28T01:10:19.7790850Z py27 installed: DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support,atomicwrites==1.3.0,attrs==19.3.0,certifi==2019.11.28,chardet==3.0.4,configparser==4.0.2,contextlib2==0.6.0.post1,freezegun==0.3.15,funcsigs==1.0.2,idna==2.9,importlib-metadata==1.5.0,mock==3.0.5,more-itertools==5.0.0,packaging==20.1,pathlib2==2.3.5,pluggy==0.13.1,py==1.8.1,pyparsing==2.4.6,pytest==4.6.9,python-dateutil==2.8.1,requests==2.23.0,scandir==1.10.0,six==1.14.0,urllib3==1.25.8,wcwidth==0.1.8,zipp==1.2.0
2020-02-28T01:10:19.7795701Z py27 run-test-pre: PYTHONHASHSEED='3292039813'
2020-02-28T01:10:19.7796265Z py27 run-test: commands[0] | py.test
2020-02-28T01:10:20.2531722Z ============================= test session starts ==============================
2020-02-28T01:10:20.2532468Z platform linux2 -- Python 2.7.17, pytest-4.6.9, py-1.8.1, pluggy-0.13.1
2020-02-28T01:10:20.2533020Z cachedir: .tox/py27/.pytest_cache
2020-02-28T01:10:20.2533388Z rootdir: /home/runner/work/openQA-python-client/openQA-python-client
2020-02-28T01:10:20.4473469Z collected 15 items
2020-02-28T01:10:20.4473548Z 
2020-02-28T01:10:20.8115094Z tests/test_client.py ...............                                     [100%]
2020-02-28T01:10:20.8115567Z 
2020-02-28T01:10:20.8115679Z ========================== 15 passed in 0.56 seconds ===========================
2020-02-28T01:10:20.8318523Z ___________________________________ summary ____________________________________
2020-02-28T01:10:20.8318657Z   py27: commands succeeded
2020-02-28T01:10:20.8318744Z   congratulations :)
2020-02-28T01:10:20.8447583Z Cleaning up orphan processes

which is what makes me think the problem is running an environment that's not in envlist...

[AdamWill]

so in the end I sort of went a different way and just put the CI environments in the envlist and I'll work around the issues with distro packaging on that end. But I'm still curious about this.

[ymyzk]

Hi, thanks for sharing your usage. The difficulty comes from the fact that tox recognize py*-ci environments when running tests but doesn't recognize when showing a list of environments because they're not explicitly defined in tox.ini.

When the following tox.ini is given:

[tox]
envlist = py{27,36,37,38,39}

[gh-actions]
python =
    2.7: py27
    3.6: py36-ci
    3.7: py37-ci
    3.8: py38-ci
    3.9: py39-ci

[testenv]
deps =
    -r{toxinidir}/install.requires
    -r{toxinidir}/tests.requires
    ci: -r{toxinidir}/ci.requires

commands =
    py.test
    ci: py.test --cov-report term-missing --cov-report xml --cov openqa_client
    ci: diff-cover coverage.xml --fail-under=90
    ci: diff-quality --violations=pylint --fail-under=90
setenv =
    PYTHONPATH = {toxinidir}

We cannot see the environment like py*-ci in the list of all available environments:

$ tox -v --listenvs-all
using tox.ini: /home/yusuke/Development/github.com/ymyzk/tox-gh-actions/openQA-python-client/tox.ini (pid 115376)
using tox-3.14.5 from /home/yusuke/Development/github.com/ymyzk/tox-gh-actions/venv/lib/python3.8/site-packages/tox/__init__.py (pid 115376)
default environments:
py27 -> [no description]
py36 -> [no description]
py37 -> [no description]
py38 -> [no description]
py39 -> [no description]

But we can see py*-ci when we run the environment.

$ tox -e py38-ci
GLOB sdist-make: /home/yusuke/Development/github.com/ymyzk/tox-gh-actions/openQA-python-client/setup.py
py38-ci create: /home/yusuke/Development/github.com/ymyzk/tox-gh-actions/openQA-python-client/.tox/py38-ci
py38-ci installdeps: -r/home/yusuke/Development/github.com/ymyzk/tox-gh-actions/openQA-python-client/ci.requires, -r/home/yusuke/Development/github.com/ymyzk/tox-gh-actions/openQA-python-client/install.requires, -r/home/yusuke/Development/github.com/ymyzk/tox-gh-actions/openQA-python-client/tests.requires

[ymyzk]

One solution is specifying a list of environments explicitly when running tox on GitHub Actions:

[tox]
envlist = py{27,36,37,38,39}

[gh-actions]
python =
    2.7: py27
    3.6: py36-ci
    3.7: py37-ci
    3.8: py38-ci
    3.9: py39-ci

[testenv]
deps =
    -r{toxinidir}/install.requires
    -r{toxinidir}/tests.requires
    ci: -r{toxinidir}/ci.requires

commands =
    py.test
    ci: py.test --cov-report term-missing --cov-report xml --cov openqa_client
    ci: diff-cover coverage.xml --fail-under=90
    ci: diff-quality --violations=pylint --fail-under=90
setenv =
    PYTHONPATH = {toxinidir}

jobs:
  build:
    steps:
    - name: Test with tox
      run: tox -e py36-ci,py37-ci,py38-ci,py39-ci

[AdamWill]

thanks for looking into it! yeah, I noticed the same thing about tox not listing the environment. it's quite hard to be sure exactly what behaviour tox 'intends' with regard to environment definitions, really - I don't know if this kind of 'implicit' environment is even really intended to work or if it's just a sort of side effect.

[ymyzk]

I don't know if this kind of 'implicit' environment is even really intended to work or if it's just a sort of side effect.

Me neither. It may be documented in the tox's documentation but I couldn't find it so far. Sadly, this is something difficult to fix/support by changing this plugin's code. Do you mind closing this ticket?

[AdamWill]

no, if it's too difficult to fix that's fine. thanks for looking into it.

[bittner]

I really think tox-gh-actions does a bad job stopping users from doing things that are against its design.

The python key in the [gh-actions] section should probably never list environments. This is what Tox is made for. Instead, tox-gh-actions should error out (or ignore the list item printing out a warning) if it doesn't define a valid factor.

[gh-actions]
python =
    2.7: py27
    3.6: py36-ci   ; <-- this line is invalid, the plugin should abort
    ...

CI Plugins should be dumb

Already in the old days of tox-travis that plugin was meant to turn an execution of tox into executing a single environment. It's like the plugin injects a TOXENV environment variable with the factors it can find (and combine, e.g. when a DJANGO environment variable is defined in addition). It's nothing but a concise alternative to writing the verbose tox -e py38, tox -e py39, etc. in your CI configuration.

CI plugins like tox-travis and tox-gh-actions should be dumb. As dumb as possible, for what it makes sense. They should only run Tox, based on simple rules that populate TOXENV. A dumb implementation will be able to detect which Python version is installed (i.e. it is being executed with); it will need help though, e.g., with Django versions.

If you want more you should configure Tox itself, not the plugin for GHA.