“Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.”
def add(a, b):
"""Add two values."""
return a + bHow do we know that this is correct?
We can go to the REPL and run the code! For example, we know that
python3and then:
>>> add(2, 2)
4Seems like our function is doing the right thing!
We can test whether a script is running and producing the correct output:
def add(a, b):
return a + b
def mul(a, b):
return a * b
def predict(x):
intercept = 5
coef = 0.7
return add(
mul(coef, x),
intercept
)def main():
for x in range(10):
y = predict(x)
print(f"f({x}) = {y}")
if __name__ == "__main__":
main()python3 model.pyThe results will be displayed on the terminal and we can check whether the results are correct or plausible by hand.
f(0) = 5.0
f(1) = 5.7
f(2) = 6.4
f(3) = 7.1
f(4) = 7.8
f(5) = 8.5
f(6) = 9.2
f(7) = 9.899999999999999
f(8) = 10.6
f(9) = 11.3- As projects grow, manually re-testing every script is tedious and error prone.
- We forget manual tests that we have done in the past and don’t do it again
- We are testing a very limited set of inputs
- Bugs may only appear for certain edge cases but we are not really searching for those
- Set up any needed data or state.
- Run the code you want to test.
- Assert the results are what you expect.
- This is only a folder structure to showcase a basic, minimal pytest example.
- This is NOT an example of how a project SHOULD be organised.
- It is purely intended to demonstrate how pytest can be made functional.
[tool.pytest.ini_options]
minversion = "7.0"
testpaths = "src/tests"
log_cli_level = "INFO"
xfail_strict = true
addopts = [
"-ra",
"--strict-config",
"--strict-markers",
"-vvv",
]We can run these tests all at once using pytest which will tell us if
they pass or fail.
pip install pytest
pytestdef add(a, b):
return a + b
def mul(a, b):
return a * bdef test_add():
result = add(2, 2)
assert result == 4
def test_mul():
result = mul(2, 2)
assert result == 4
def add(a, b):
return a - bpytest
FAIL
pip install pytest-covFrom the repository root run:
pytest --cov=src
From the repository root run:
pytest --cov=src --cov-report term-missing
If you have lots of tests and they take some time you can use
pytest-xdist which will run tests in parallel when you use the -n
flag:
pip install pytest-xdist
pytest --cov=src --cov-report term-missing -n 16Note, that in our example this will run slower since we only have two functions and the parallelising overhead is not really worth it. But if your tests are running longer than a few seconds, this will likely already be worth it.
def add(a, b):
return a + b
def mul(a, b):
return a * bdef test_add():
result = add(2, 2)
assert result == 4
def test_mul():
result = mul(2, 2)
assert result == 4def add(a, b):
return a * b
def mul(a, b):
return a + bdef test_add():
result = add(2, 2)
assert result == 4
def test_mul():
result = mul(2, 2)
assert result == 4The behaviour of the functions has completely changed, but the tests will pass, so we might think all is well! (Remember the quote from the beginning?)
This is precisely what mutation testing does: It takes your code and creates mutated variants of your code. It will then run your tests to see if your tests are good enough to catch the mutation. If your tests still pass, you might have to improve your testing! In this example it may be enough to simply add a few more test cases:
def add(a, b):
return a * bdef test_add():
result = add(2, 2)
assert result == 4
result = add(7, 2)
assert result == 9
result = add(1, 2)
assert result == 3| Name/Link | Pros | Cons |
|---|---|---|
| mutmut | Actively maintained, can cache previous progress | Not written by me |
| pymute | Written by me | Can’t cache previous progress (yet) |
| mut.py | Couldn’t find any | Not actively maintained, not compatible with current versions, do not use |
| mut.py fork | Was forked because mut.py was not actively maintained | Also not actively maintained |
- Previously we called our function multiple times with different inputs
- Pytest allows us something nicer, by parametrising test functions using a decorator
@pytest.mark.parametrize(
"a,b,expected_result",
[(2, 2, 4), (1, 2, 3), (5, 5, 10), (2, 4, 6), (100, 100, 200)],
)
def test_add(a, b, expected_result):
result = add(a, b)
assert result == expected_result@pytest.mark.parametrize(
"a,b,expected_result",
[(2, 2, 4), (1, 2, 2), (5, 5, 25), (2, 4, 8), (100, 100, 10000)],
)
def test_mul(a, b, expected_result):
result = mul(a, b)
assert result == expected_result
We want to test that this function raises an error if the input is in a bad state:
def add(a, b):
if isinstance(a, str) or isinstance(b, str):
raise ValueError("Adding of strings not allowed!")
return a + bWe can use parametrize to create multiple inputs with bad states, and
pytest.raises to assert that the correct error is raised:
@pytest.mark.parametrize("a,b", [(5, "6"), ("5", 6), ("5", "6")])
def test_add_error(a, b):
with pytest.raises(ValueError, match="Adding of strings not allowed!"):
add(a, b)
Imagine that in some commit for whatever reason we accidentally change this:
def add(a, b):
if isinstance(a, str) or isinstance(b, str):
raise ValueError("Adding of strings not allowed!")
return a + bTo this:
def add(a, b):
if isinstance(a, str):
raise ValueError("Adding of strings not allowed!")
return a + bOur tests will then fail when b is in a bad state, since we don’t get
the error that we expected:
def test_add_array():
a = np.arange(9).reshape(3, 3).astype(float)
b = np.arange(10, 19).reshape(3, 3).astype(float)
result = add(a, b)
expected_result = np.array(
[[10.0, 12.0, 14.0], [16.0, 18.0, 20.0], [22.0, 24.0, 26.0]],
)
assert result == pytest.approx(expected_result)import sys
from pathlib import Path
import pytest
import numpy as np
src = Path(__file__).parents[0] / ".." / "."
print(src)
sys.path.append(str(src))
from model import add, mul, predict- if you have a medium/large sized folder of files with code that is shared by different executables, you probably want to properly package it (by that I only mean making the local project installable via pip, NOT necessarily publishing on PyPI)
- python/pytest
importf***ery can be quite confusing - if your project is packaged and you want to specifically test the
installed version rather than local modules available in your
$PYTHONPATH, you may want to look into using tox - The main point here is that you do NOT need to properly package your project in order to write and run tests with pytest. You can do this easily with any project you already have that defines any functions or classes you can import elsewhere.