A mutation testing tool for Go
Gremlins is a mutation testing tool for Go. It has been made to work well on smallish Go modules, for example microservices, on which it helps validate the tests, aids the TDD process and can be used as a CI quality gate. As of now, Gremlins doesn't work very well on very big Go modules, mainly because a run can take hours to complete.
Gremlins is still in its 0.x.x release, which, as per SemVer, doesn't guarantee backward compatibility. What this means for Gremlins is that configuration flags and/or configuration files can change among minor releases. This may be important if you use Gremlins in automated builds, for example in continuous integration pipelines.
Also, while in 0.x.x releases, only the current minor release will be maintained; if we are in v0.2.0, no patch releases will be provided for the v0.1.0 line. Once Gremlins will reach it's 1.x.x release, it will start to be backward compatible, and we'll decide how many releases to maintain at the same time (ex. up to two minors before, as Go itself does).
Code coverage is unreliable as a measure of test quality. It is too easy to have tests that exercise a piece of code but don't test anything at all.
Mutation testing works by mutating the code exercised by the tests and verifying if the mutation is caught by the test suite. Imagine gremlins going into your code and messing around: will your test suit catch their damage?
Here is a nice intro to mutation testing.
Please refer to the documentation for instructions on how to obtain, configure and use Gremlins.
This is just to get you started, do not forget to check the complete documentation.
Download the pre-built binary for your OS/ARCH from
the release page
and put it somewhere in the PATH, then:
gremlins unleashGremlins will report each mutation as:
RUNNABLE: In dry-run mode, a mutation that can be tested.NOT COVERED: A mutation not covered by tests; it will not be tested.KILLED: The mutation has been caught by the test suite.LIVED: The mutation hasn't been caught by the test suite.TIMED OUT: The tests timed out while testing the mutation: the mutation actually made the tests fail, but not explicitly.NOT VIABLE: The mutation makes the build fail.
Mutation testing exists since the early days of computer science, so loads of papers and articles do exists. Gremlins is inspired from those.
Among the existing mutation testing tools, Gremlins is inspired especially by PITest.
There is not much around, except from:
See contributing.
Gremlins is licensed under the Apache 2.0 License. The Gremlins documentation is licensed under CC BY-SA 4.0.