TravisCI and Tox
Updated:
I recently had the displeasure of configuring TravisCI
to work with tox
.
TravisCI
has a “build matrix” system, where you can configure different environment conditions to test your code.
Oh hey, that sounds just like tox
you said. Yup.
However, tox
runs anywhere.
I don’t get to develop on TravisCI
machines.
So we have to support both systems in a crude manner, at least until TravisCI
has better native support.
Tox environments
I use tox environments to test pytest-antilru under different conditions:
- Python 2.7/3.5/3.6/3.7
backports.functools_lru_cache
functools32.lru_cache
This is done using environment factors, as demonstrated in pytest-antilru tox.ini:
[tox]
envlist = py{27}-{backports,functools32}-pytest{2,3,4},py{35,36,37}-pytest{2,3,4},project_tests
These work like bash brace expansion, which results in a cartesian product. Tox documentation has more details.
What results is an expanded list of many combinations:
$ tox -l
py27-backports-pytest2
py27-backports-pytest3
py27-backports-pytest4
py27-functools32-pytest2
py27-functools32-pytest3
py27-functools32-pytest4
py35-pytest2
py35-pytest3
py35-pytest4
py36-pytest2
py36-pytest3
py36-pytest4
py37-pytest2
py37-pytest3
py37-pytest4
project_tests
TravisCI Build Matrix
TravisCI
supports a very similar idea, called build matrix.
Many of the config parameters accept lists, which expands in cartesian product manner.
When different python version is set, TravisCI
is using pyenv
internally to switch.
We could use this knowledge to our benefit but it’s real janky.
Instead, we should play nice and instead have TravisCI
invoke only the applicable tests for the python version.
Unfortunately, tox
doesn’t support globbing on environments.
We can’t simply tell TravisCI
to run all py27
factors, we need to explicitly pass every single env to tox
.
Ugh.
Enter this mess that I stole from flake8. I think mine is a little bit more readable (I try to avoid too much bash magic).
script:
# Travis only supports one python version per environment (pyenv).Tox doesn't support running all permutations of a
# factor. They found each other, though the magic of grep and sed.
- export PY_VERSION=$(echo "${TRAVIS_PYTHON_VERSION}" | sed 's/\.//')
- export TOXENV=$(tox --listenvs | grep "py${PY_VERSION}-" | tr '\n' ',')
- tox
- We first get the python version from environment. This corresponds to ‘3.5’ or ‘3.7-dev’ strings.
- Some string manipulation to format it to
tox
python version format (two-letter value). - Grep through the possible tox environments for the ones that apply.
- More gross string manipulation because tox only accepts comma-delimited string.
- Set
TOXENV
.tox
reads from several places to get the environment list.
With all this set in place, TravisCI
generates one test runner for each python version in parallel.
It’s nice because failures will be isolated and easily identified from TravisCI
build email.
And separate logs is nice.
But I really wish these two would play better with each other, more native support.