Automating Python Multi-Version Testing With Tox and Nox

In modern Python development, maintaining compatibility across multiple Python versions is super critical, especially for libraries and tools that target a diverse user base. Here, we explore how to use Tox and Nox, two powerful tools for Python test automation, to validate projects across multiple Python versions. Using a concrete project as an example, we’ll walk through setting up multi-version testing, managing dependencies with Poetry, and using Pytest for unit testing.

Why Automate Multi-Version Testing?

Automating tests across Python versions ensures your project remains robust and reliable in diverse environments. Multi-version testing can:

• Catch compatibility issues early.
• Provide confidence to users that your package works on their chosen Python version.
• Streamline the development process by reducing manual testing.

What Are Tox and Nox?

• Tox: A well-established tool designed to automate testing in isolated environments for multiple Python versions. It is declarative, with configurations written in an ini file, making it easy to set up for standard workflows.
• Nox: A more flexible alternative that uses Python scripts for configuration, allowing dynamic and programmable workflows. It is based on a python code config file, so it is more robust than Tox to complex project configurations.

Both tools aim to simplify testing across multiple Python versions, but their approaches make them suited for different needs.

Project Setup: A Practical Example

Let’s set up a project named tox-nox-python-tests to demonstrate the integration of Tox, Nox, Poetry, and Pytest. The project includes simple addition and subtraction functions, tested across Python versions 3.8 to 3.13.

1. Directory Structure

Here’s how the project is structured:

tox-nox-python-tests/
├── tox_nox_python_tests/
│   ├── __init__.py
│   ├── main.py
│   └── calculator.py
├── tests/
│   ├── __init__.py
│   └── test_calculator.py
├── pyproject.toml
├── tox.ini
├── noxfile.py
├── README.md

2. Core Functionality

The calculator module contains two basic functions:

calculator.py

def add(a, b):
    """Returns the sum of two numbers."""
    return a + b

def subtract(a, b):
    """Returns the difference of two numbers."""
    return a - b

We will use pytest for unit testing (with parametrized tests):

test_calculator.py

import pytest
from tox_nox_python_tests.calculator import add, subtract

@pytest.mark.parametrize("a, b, expected", [
    (1, 2, 3),
    (-1, 1, 0),
    (0, 0, 0),
])
def test_add(a, b, expected):
    assert add(a, b) == expected

@pytest.mark.parametrize("a, b, expected", [
    (5, 3, 2),
    (10, 5, 5),
    (-1, -1, 0),
])
def test_subtract(a, b, expected):
    assert subtract(a, b) == expected

3. Dependency Management With Poetry

Poetry manages dependencies and environments. The pyproject.toml file is the modern way to manage Python projects and replaces traditional setup.py and setup.cfg files:

pyproject.toml

[tool.poetry]
name = "tox_nox_python_tests"
version = "0.1.0"
description = "Automate Python testing across multiple Python versions using Tox and Nox."
authors = ["Wallace Espindola <wallace.espindola@gmail.com>"]
readme = "README.md"
license = "MIT"

[tool.poetry.dependencies]
python = "^3.8"
pytest = "^8.3"

[build-system]
requires = ["poetry-core>=1.8.0"]
build-backend = "poetry.core.masonry.api"

Run the following commands to install dependencies and create a virtual environment:

poetry install

4. Running Unit Tests With Pytest

 You can run the basic unit tests this way:

poetry run pytest --verbose

5. Multi-Version Testing With Tox

Tox automates testing across Python versions using isolated virtual environments. The configuration is declarative and resides in a tox.ini file, testing the app with Python versions from 3.8 to 3.13:

tox.ini

[tox]
envlist = py38, py39, py310, py311, py312, py313

[testenv]
allowlist_externals = poetry
commands_pre =
    poetry install --no-interaction --no-root
commands =
    poetry run pytest --verbose

Run Tox using:

poetry run tox

Tox will automatically create environments for each Python version and execute the pytest suite in isolation.

Example output:

6. Flexible Testing With Nox

For more dynamic configurations, use Nox. Its Python-based scripts allow complex logic and custom workflows and test the app with Python versions from 3.8 to 3.13:

noxfile.py

import nox

@nox.session(python=["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"])
def tests(session):
    session.install("poetry")
    session.run("poetry", "install", "--no-interaction", "--no-root")
    session.run("pytest")

Run Nox using:

poetry run nox

Nox offers flexibility for customizing workflows, such as conditional dependency installation or environment-specific configurations.

Example output:

7. Integration With CI/CD

Automate the testing process further by integrating Tox and/or Nox into a CI/CD pipeline. 

For example, a GitHub Actions workflow can look like this:

.github/workflows/python-tests.yml

name: Python Tests

on:
  push:
    branches:
      - main
  pull_request:

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.8, 3.9, 3.10, 3.11, 3.12, 3.13]
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
      - name: Install Poetry
        run: pip install poetry
      - name: Install Dependencies
        run: poetry install --no-interaction --no-root
      - name: Run Tests with Tox
        run: poetry run tox
      - name: Run Tests with Nox
        run: poetry run nox

As an option, the following GitLab CI/CD pipeline runs tests across multiple Python versions:

.gitlab-ci.yml

stages:
  - test

tox-tests:
  stage: test
  image: python:${PYTHON_VERSION}
  variables:
    PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
  before_script:
    - pip install poetry
    - poetry install --no-interaction --no-root
  script:
    - poetry run tox
  cache:
    paths:
      - .tox/
      - .cache/pip/
  parallel:
    matrix:
      - PYTHON_VERSION: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]

This configuration uses the matrix keyword to run tests across specified Python versions in parallel. 

Keep in mind that you run either Tox or Nox in your project, not both, since they have similar goals.

Conclusion

By combining Tox or Nox with Poetry and Pytest, you can achieve seamless multi-version test automation for Python projects. Whether you prefer the declarative approach of Tox or the programmable flexibility of Nox, these tools ensure your code is validated across a wide range of Python environments. That approach is especially useful for shared libs projects and multi-user environments projects.

For the complete example, check the project repository GitHub: tox-nox-python-tests.

References

This project uses Tox, Nox, Poetry, and Pytest for test automation. For detailed documentation, refer to:

• Tox Documentation
• Nox Documentation
• Poetry Documentation
• Pytest Documentation