Merge pull request #49 from darrenburns/test-annotation

Descriptive testing

README.md

@@ -5,12 +5,15 @@
 
 A modern Python test framework designed to help you find and fix flaws faster.
 
-![screenshot](https://raw.githubusercontent.com/darrenburns/ward/master/screenshot.png)
+![screenshot](screenshot.png)
 
 ## Features
 
 This project is a work in progress. Some of the features that are currently available in a basic form are listed below.
 
+* **Descriptive test names:** describe what your tests do using strings, not function names.
+* **Powerful test selection:** limit your test run not only by matching test names/descriptions, but also on the code 
+contained in the body of the test.
 * **Colourful, human readable output:** quickly pinpoint and fix issues with detailed output for failing tests.
 * **Modular test dependencies:** manage test setup/teardown code using modular pytest-style fixtures.
 * **Expect API:** A simple but powerful assertion API inspired by [Jest](https://jestjs.io).
@@ -34,20 +37,20 @@ Install Ward with `pip install ward`.
 Write your first test in `test_sum.py` (module name must start with `"test"`):
 
 ```python
-from ward import expect
+from ward import expect, test
 
-def test_one_plus_two_equals_three():  # name must start with "test"
+@test("1 plus 2 equals 3")
+def _():
     expect(1 + 2).equals(3)
 ```
 
 Now run your test with `ward` (no arguments needed). Ward will output the following:
 
 ```
- PASS  test_sum.test_one_plus_two_equals_three
+ PASS  test_sum: 1 plus 2 equals 3
 ```
 
-*You've just wrote your first test with Ward, congrats!* Look [here](#more-examples) for more examples of
-how to test things with Ward.
+*You've just wrote your first test with Ward, congrats!* Look [here](#more-examples) for more examples.
 
 ## How to Contribute
 
@@ -57,6 +60,101 @@ See the [contributing guide](.github/CONTRIBUTING.md) for information on how you
 
 ## More Examples
 
+### Descriptive testing
+
+Test frameworks usually require that you describe how your tests work using
+a function name. As a result test names are often short and non-descriptive,
+or long and unreadable.
+
+Ward lets you describe your tests using strings, meaning you can be as descriptive
+as you'd like:
+
+```python
+from ward import expect, test
+
+NAME = "Win Butler"
+
+@test("my_sum(1, 2) is equal to 3")
+def _():
+    total = my_sum(1, 2)
+    expect(total).equals(3)
+
+@test(f"first_char('{NAME}') returns '{NAME[0]}'")
+def _():
+    first_char = first_char(NAME)
+    expect(first_char).equals(NAME[0])
+```
+
+During the test run, Ward will print the descriptive test name to the console:
+
+```
+FAIL  test_things: my_sum(1, 2) is equal to 3
+PASS  test_things: first_char('Win Butler') returns 'W'
+```
+
+If you'd still prefer to name your tests using function names, you can do so
+by starting the name of your test function with `test_`:
+
+```python
+def test_my_sum_returns_the_sum_of_the_input_numbers():
+    total = my_sum(1, 2)
+    expect(total).equals(3)
+```
+
+### Test selection 
+
+#### Search and run matching tests with `--search`
+
+You can choose to limit which tests are collected and ran by Ward 
+using the `--search STRING` option. Test names, descriptions *and test function bodies*
+will be searched, and those which contain `STRING` will be ran. Here are
+some examples:
+
+**Run all tests that call the `fetch_users` function:**
+```
+ward --search "fetch_users("
+```
+
+**Run all tests that check if a `ZeroDivisionError` is raised:**
+```
+ward --search "raises(ZeroDivisionError)"
+```
+
+**Run all tests decorated with the `@xfail` decorator:**
+```
+ward --search "@xfail"
+```
+
+**To run a test called `test_the_sky_is_blue`:**
+
+```text
+ward --search test_the_sky_is_blue
+```
+
+**Running tests inside a module:**
+
+The search takes place on the fully qualified name, so you can run a single
+module (e.g. `my_module`) using the following command:
+
+```text
+ward --search my_module.
+```
+
+Of course, if a test name or body contains the string `"my_module."`, that test
+will also be selected and ran. 
+
+This approach is useful for quickly querying tests and running those which match a
+simple query, making it useful for development.
+
+Of course, sometimes you want to be very specific when declaring which tests to run.
+
+#### Specific test selection
+
+Ward will provide an option to query tests on name and description using substring
+or regular expression matching.
+
+(TODO)
+
 ### Dependency injection with fixtures
 
 In the example below, we define a single fixture named `cities`.
@@ -65,34 +163,42 @@ Ward sees that the fixture name and parameter names match, so it
 calls the `cities` fixture, and passes the result into the test.
 
 ```python
-from ward import expect, fixture
+from ward import test, expect, fixture
 
 @fixture
 def cities():
     return ["Glasgow", "Edinburgh"]
 
-def test_using_cities(cities):
-    expect(cities).equals(["Glasgow", "Edinburgh"])
+@test("'Glasgow' should be contained in the list of cities")
+def _(cities):
+    expect("Glasgow").contained_in(cities)
 ```
 
+The fixture will be executed each time it gets injected into a test.
+
 Fixtures are great for extracting common setup code that you'd otherwise need to repeat at the top of your tests, 
 but they can also execute teardown code:
 
 ```python
+from ward import test, expect, fixture
+
 @fixture
 def database():
     db_conn = setup_database()
     yield db_conn
     db_conn.close()
 
 
-def test_database_connection(database):
+@test(f"Bob is one of the users contained in the database")
+def _(database):
     # The database connection can be used in this test,
     # and will be closed after the test has completed.
     users = get_all_users(database)
     expect(users).contains("Bob")
 ```
 
+The code below the `yield` statement in a fixture will be executed after the test runs. 
+
 ### The `expect` API
 
 Use `expect` to perform tests on objects by chaining together methods. Using `expect` allows Ward
@@ -123,10 +229,11 @@ and more. If a test fails due to the mock not being used as expected, Ward will
 debugging the problem.
 
 ```python
-from ward import expect
+from ward import test, expect
 from unittest.mock import Mock
 
-def test_mock_was_called():
+@test("the mock was called with the expected arguments")
+def _():
     mock = Mock()
     mock(1, 2, x=3)
     expect(mock).called_once_with(1, 2, x=3)
@@ -138,8 +245,9 @@ The test below will pass, because a `ZeroDivisionError` is raised. If a `ZeroDiv
 the test would fail.
 
 ```python
-from ward import raises
+from ward import raises, test
 
+@test("a ZeroDivision error is raised when we divide by 0")
 def test_expecting_an_exception():
     with raises(ZeroDivisionError):
         1/0
@@ -157,26 +265,6 @@ ward --path tests
 To run tests in the current directory, you can just type `ward`, which
 is functionally equivalent to `ward --path .`
 
-
-### Filtering tests by name
-
-You can choose to limit which tests are collected and ran by Ward 
-using the `--filter` option. Test names which contain the argument value 
-as a substring will be run, and everything else will be ignored.
-
-To run a test called `test_the_sky_is_blue`:
-
-```text
-ward --filter test_the_sky_is_blue
-```
-
-The match takes place on the fully qualified name, so you can run a single
-module (e.g. `my_module`) using the following command:
-
-```text
-ward --filter my_module.
-```
-
 ### Skipping a test
 
 Use the `@skip` annotation to tell Ward not to execute a test.
@@ -186,7 +274,23 @@ from ward import skip
 
 @skip
 def test_to_be_skipped():
-    pass
+    # ...
+```
+
+You can pass a `reason` to the `skip` decorator, and it will be printed
+next to the test name/description during the run.
+
+```python
+@skip("not implemented yet")
+@test("everything is okay")
+def _():
+    # ...
+```
+
+Here's the output Ward will print to the console when it runs the test above:
+
+```
+SKIP  test_things: everything is okay  [not implemented yet]
 ```
 
 ### Expecting a test to fail

setup.py

@@ -3,7 +3,7 @@
 with open("README.md", "r") as fh:
     long_description = fh.read()
 
-version = "0.10.0a0"
+version = "0.11.0a0"
 
 setup(
     name="ward",
@@ -18,5 +18,10 @@
     packages=["ward"],
     python_requires=">=3.6",
     entry_points={"console_scripts": ["ward=ward.run:run"]},
-    install_requires=["colorama==0.4.1", "termcolor==1.1.0", "dataclasses==0.6", "click==7.0"],
+    install_requires=[
+        "colorama==0.4.1",
+        "termcolor==1.1.0",
+        "dataclasses==0.6",
+        "click==7.0",
+    ],
 )

tests/test_collect.py

@@ -0,0 +1,57 @@
+from modulefinder import ModuleFinder
+from pkgutil import ModuleInfo
+
+from ward import test, fixture, expect, raises
+from ward.collect import search_generally, is_test_module
+from ward.testing import Test
+
+
+def named():
+    expect("fox").equals("fox")
+
+
+@fixture
+def named_test():
+    return Test(fn=named, module_name="my_module")
+
+
+@fixture
+def tests_to_search(named_test):
+    return [named_test]
+
+
+@test("search_generally matches on qualified test name")
+def _(tests_to_search, named_test):
+    results = search_generally(tests_to_search, query="my_module.named")
+    expect(list(results)).equals([named_test])
+
+
+@test("search_generally matches on test name alone")
+def _(tests_to_search, named_test):
+    results = search_generally(tests_to_search, query="named")
+    expect(list(results)).equals([named_test])
+
+
+@test("search_generally query='fox' returns tests with 'fox' in the body")
+def _(tests_to_search, named_test):
+    results = search_generally(tests_to_search, query="fox")
+    expect(list(results)).equals([named_test])
+
+
+@test("search_generally returns an empty generator when no tests match query")
+def _(tests_to_search):
+    results = search_generally(tests_to_search, query="92qj3f9i")
+    with raises(StopIteration):
+        next(results)
+
+
+@test("is_test_module returns True when module name begins with 'test_'")
+def _():
+    module = ModuleInfo(ModuleFinder(), "test_apples", False)
+    expect(is_test_module(module)).equals(True)
+
+
+@test("is_test_module returns False when module name doesn't begin with 'test_'")
+def _():
+    module = ModuleInfo(ModuleFinder(), "apples_test", False)
+    expect(is_test_module(module)).equals(False)