Tests4Py is an easy-to-use benchmark inspired by (BugsInPy)[https://github.com/soarsmu/BugsInPy] for evaluating and analyzing all things testing and debugging related. Each subject in Tests4Py comes with a testing oracle, predefined system and unit tests, and the possibility to generate new system and unit tests on demand.
Tests4Py requires pyenv to leverage the correct Python version for each subject. Tests4Py will automatically install pyenv if it is not installed on your machine. If this process fails, you can install pyenv manually by following the instructions at https://github.com/pyenv/pyenv#installation.
Installing Tests4Py is as easy as calling the Python package manager pip:
pip install tests4pyIf you have separate Python 2 and Python 3 on your machine, you may need to run the following:
pip3 install tests4pyIf you want to install the latest development version, you can install Tests4Py directly from GitHub:
pip install git+https://github.com/smythi93/Tests4PyThe API of Tests4Py provides several commands, making it easy to use the benchmark. You can leverage the API by importing Tests4Py in your program (see (this notebook)[https://github.com/smythi93/Tests4Py] for further information). Each call to the API returns an instance of a report that provides the information about the executed command. Furthermore, each of these commands is accessible via Tests4Py's CLI. You can find a listing of Tests4Py's most important commands in the following (additionally, you can try it yourself with (this notebook)[https://github.com/smythi93/Tests4Py]).
t4p is Tests4Py's CLI. It provides all commands to interact with Tests4Py. You can get a list of all available
commands by calling t4p without any arguments or t4p --help for a detailed list.
usage: t4p [-h] [--report REPORT] [--debug] [--version] {info,checkout,build,test,unittest,systemtest,config,cache,clear,grammar,sfl,run} ...
The access point to the tests4py framework
positional arguments:
{info,checkout,build,test,unittest,systemtest,config,cache,clear,grammar,sfl,run}
The command of the tests4py framework to execute
info Get information of this benchmark and its projects
checkout Check out a project
build Build a project
test Run tests on a project that are included in the project
unittest Run or generate unittests
systemtest Run or generate system tests
config Edit the configuration of this benchmark
cache Cache a project to you local machine for faster access
clear Clear cached projects from your local machine
grammar Get the grammar of a project that specifies the input format
sfl Conduct sfl (statistical fault localization) on a project
run Run an input on a project and observe its result
options:
-h, --help show this help message and exit
--report REPORT Output the report to a file
--debug Activate debugging log
--version Show version number
The info command provides all information about Tests4Py, its projects, and all its subjects.
It is accessible via the CLI with the following:
t4p info [-h] [-p PROJECT_NAME] [-i BUG_ID]
options:
-h, --help show this help message and exit
-p PROJECT_NAME The id of the project for which the information shall be printed
-i BUG_ID The bug number of the project_name for which the information shall be printed
The checkout command retrieves the source code of a subject to a defined position. Test4Py provides built-in caching;
as soon as you check out a subject, Tests4Py will store the project repository in your home directory and uses this
version for further checkouts.
t4p checkout [-h] -p PROJECT_NAME -i BUG_ID [-r] [-w WORK_DIR] [-u] [-f]
options:
-h, --help show this help message and exit
-p PROJECT_NAME The name of the project for which a particular version shall be checked out. Run tests4py info to
check available project
-i BUG_ID The number of bug from project in tests4py. Run tests4py info to check bug id number
-r, --repaired Set the flag to checkout the repaired version, without the flag the buggy version will be checked out
-w WORK_DIR The working directory to which the buggy or fixed project version shall be checked out. The working
directory has to be either empty or a previously used working directory. Default will be (./tmp)
-u If set the project won't be checked out again if it already exists at the specified location but
only the tests4py data will be updated
-f If set the command won't use any cached version, even if the global cache flag is set
The build command utilizes the correct Python version based on the subject and your machine leveraging
pyenv, establishes the virtual environment, including all dependencies, and installs
the subject in this environment. Tests4Py will store the virtual environment in your home directory and uses this
environment for further compiles of this installed subject.
t4p build [-h] [-w WORK_DIR] [-r] [-f]
options:
-h, --help show this help message and exit
-w WORK_DIR The working directory to build the project. Default will be the current directory
-r Set to rebuild the project from scratch
-f If set the command won't use any cached version, even if the global cache flag is set
The test command executes the unit tests included in the project and outputs their results.
t4p test [-h] [-w WORK_DIR] [-t SINGLE_TEST] [-r] [-a] [-o OUTPUT]
options:
-h, --help show this help message and exit
-w WORK_DIR The working directory to run the test. Default will be the current directory
-t SINGLE_TEST Run single test from input. Default will run the test case that are failing for the bugs. Format for
pytest: <test_file_path>::<test_method>. Format for unittest:
<test_file_path_without.py>.<test_class>.<test_method>. Use tests4py info to get the information
about the project
-r Run relevant test cases for the bug in the project. Default will run the test case that are failing
for the bugs
-a Run all test cases in the project. Default will run the test case that are failing for the bugs
-o OUTPUT Output test results to file
To leverage the Tests4Py unit test capability, you must invoke the unittest with the two subcommands test that
executes unit tests or generate. The generation can create an arbitrary number of failing and passing unit tests that
can be investigated and executed.
t4p unittest [-h] {generate,test} ...
positional arguments:
{generate,test} The subcommand of the unittest to execute
generate Generate new unittests
test Run the unittests
options:
-h, --help show this help message and exit
t4p unittest test [-h] [-w WORK_DIR] [-p PATH] [-d] [-o OUTPUT]
options:
-h, --help show this help message and exit
-w WORK_DIR The working directory to run the test. Default will be the current directory
-p PATH The output path of the generated tests. Default will be (-w/tests4py_unittests.py) if -d is not set,
otherwise only the diversity tests are executed
-d Set to run diversity tests. When giving -p all tests in -p and the diversity tests are executed and
-o OUTPUT Output test results to file
t4p unittest generate [-h] [-w WORK_DIR] [-p PATH] [-n N] [-f P] [--passing] [--failing] [-a] [-v]
options:
-h, --help show this help message and exit
-w WORK_DIR The working directory to run the test. Default will be the current directory
-p PATH The output path of the generated tests. Default will be (tests4py_unittests.py)
-n N The number of generated tests. Default will be 1
-f P The number or probability of generated failing tests. If the number is a probability (<1) it will get
multiplied with -n to get the total number of failing tests. Default will be 1
--passing Set to generate only passing tests (<=> -p == 0). Cannot be set when --failing is set
--failing Set to generate only failing tests (<=> -p == -n). Cannot be set when --passing is set
-a Set to append the generated tests to the existing tests at -p
-v Set to verify the generated tests
You can also leverage the Tests4Py system test capability by invoking the systemtest command with the two subcommands
test or generate. The generation can create an arbitrary number of failing and passing system tests that can be
investigated and executed.
t4p systemtest [-h] {generate,test} ...
positional arguments:
{generate,test} The subcommand of the systemtest to execute
generate Generate new systemtests
test Run the systemtests
options:
-h, --help show this help message and exit
t4p systemtest test [-h] [-w WORK_DIR] [-p PATH] [-d] [-o OUTPUT]
options:
-h, --help show this help message and exit
-w WORK_DIR The working directory to run the test. Default will be the current directory
-p PATH The output path of the generated tests. Default will be (-w/tests4py_systemtests) if -d is not set,
otherwise only the diversity tests are executed
-d Set to run diversity tests. When giving -p all tests in -p and the diversity tests are executed and
-o OUTPUT Output test results to file
t4p systemtest generate [-h] [-w WORK_DIR] [-p PATH] [-n N] [-f P] [--passing] [--failing] [-a] [-v]
options:
-h, --help show this help message and exit
-w WORK_DIR The working directory to run the test. Default will be the current directory
-p PATH The output path of the generated tests. Default will be (tests4py_systemtests)
-n N The number of generated tests. Default will be 1
-f P The number or probability of generated failing tests. If the number is a probability (<1) it will get
multiplied with -n to get the total number of failing tests. Default will be 1
--passing Set to generate only passing tests (<=> -f == 0). Cannot be set when --failing is set
--failing Set to generate only failing tests (<=> -f == -n). Cannot be set when --passing is set
-a Set to append the generated tests to the existing tests at -p
-v Set to verify the generated tests
The run command executes an input on the subject and outputs the result of the execution or when invoking the oracle
consider it as a system test.
t4p run [-h] [-w WORK_DIR] [-o] test [test ...]
positional arguments:
test The path or string of the input
options:
-h, --help show this help message and exit
-w WORK_DIR The working directory to run the input. Default will be the current directory or last cached directory
-o Invoke the oracle to produce the result, if the input would be a test
To configure Tests4Py, you can use the config command. It allows you to set the global cache flag and the last
working directory (which will be set automatically to the last used working directory).
t4p config [-h] flag value
positional arguments:
flag The flag for setting the value
value The value to set the flag
options:
-h, --help show this help message and exit
We provide three notebooks that demonstrate the usage of Tests4Py and its capabilities on the example of statistical
fault localization. You can find them in the notebooks folder.
Feel free to try them and experiment with them, since they provide easy access to Tests4Py.
You can currently cite Tests4Py as follows:
@misc{smytzek2023tests4py,
title={Tests4Py: A Benchmark for System Testing},
author={Marius Smytzek and Martin Eberlein and Batuhan Serce and Lars Grunske and Andreas Zeller},
year={2023},
eprint={2307.05147},
archivePrefix={arXiv},
primaryClass={cs.SE}
}
Tests4Py is licensed under the MIT License. See LICENSE for the full license text.