Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework the readme #167

Merged
merged 2 commits into from
Jun 22, 2024
Merged

Rework the readme #167

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5579471
WIP: start on the README
ev-br Jun 22, 2024
3c06291
DOC: put pytest first in README; tweak it some more while at it
ev-br Jun 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
214 changes: 116 additions & 98 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ the output. Thus the example source needs to be valid python code still.
## Install and test

```
$ pip install -e .
$ pip install scipy-doctest
$ pytest --pyargs scipy_doctest
```

Expand All @@ -93,9 +93,56 @@ or nearly so.

The other layer is the `pytest` plugin.

### Run doctests via pytest

To run doctests on your package or project, follow these steps:

1. **Install the plugin**

```bash
pip install scipy-doctest
```

2. **Register or load the plugin**

Next, you need to register or load the pytest plugin within your test module or `conftest.py` file.

To do this, add the following line of code:

```python
# In your conftest.py file or test module

pytest_plugins = "scipy_doctest"
```

Check out the [pytest documentation](https://docs.pytest.org/en/stable/how-to/writing_plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file) for more information on requiring/loading plugins in a test module or `conftest.py` file.

3. **Run doctests**

Once the plugin is registered, run the doctests by executing the following command:

```bash
$ python -m pytest --doctest-modules
```
or
```bash
$ pytest --pyargs <your-package> --doctest-modules
```

By default, all doctests are collected. To only collect public objects, `strategy="api"`,
use the command flag

```bash
$ pytest --pyargs <your-package> --doctest-modules --doctest-collect=api
```

See [More fine-grained control](https://github.com/scipy/scipy_doctest#More-fine-grained-control) section
for details on how to customize the behavior.


### Basic usage

The use of `pytest` is optional, and you can use the `doctest` layer API.
For example,

```
Expand All @@ -112,7 +159,8 @@ For more details, see the `testmod` docstring. Other useful functions are
`find_doctests`, `run_docstring_examples` and `testfile` (the latter two mimic
the behavior of the eponymous functions of the `doctest` module).

#### Command-line interface

### Command-line interface

There is a basic CLI, which also mimics that of the `doctest` module:
```
Expand All @@ -127,8 +175,10 @@ Text files can also be CLI-checked:
$ python -m scipy_doctest bar.rst
```

Notice that the command-line usage only uses the default `DTConfig` settings.


#### More fine-grained control
## More fine-grained control

More fine-grained control of the functionality is available via the following
classes
Expand All @@ -147,133 +197,104 @@ configuration is simply creating an instance, overriding an attribute and
passing the instance to `testmod` or constructors of `DT*` objects. Defaults
are provided, based on a long-term usage in SciPy.

See the [DTConfig docstring](https://github.com/scipy/scipy_doctest/blob/main/scipy_doctest/impl.py#L24)
for the full set of attributes that allow you to fine-tune your doctesting experience.

### The SciPy Doctest Pytest Plugin
To set any of these attributes, create an instance of `DTConfig` and assign the attributes
in a usual way.

The pytest plugin enables the use of `scipy_doctest` tools to perform doctests.
If using the pytest plugin, it is convenient to use the default instance, which
is predefined in `scipy_doctest/conftest.py`. This instance will be automatically
passed around via an
[attribute of pytest's `Config` object](https://github.com/scipy/scipy_doctest/blob/58ff06a837b7bff1dbac6560013fc6fd07952ae2/scipy_doctest/plugin.py#L39).

Follow the given instructions to utilize the pytest plugin for doctesting.
### Examples

### Running Doctests on SciPy
```
dt_config = DTConfig()
```

1. **Install plugin**
or, if using pytest,

```bash
pip install scipy-doctest
```python
from scipy_doctest.conftest import dt_config # a DTConfig instance with default settings
```

2. **Configure Your Doctesting Experience**
and then

To tailor your doctesting experience, you can utilize an instance of `DTConfig`.
An in-depth explanation is given in the [tailoring your doctesting experience](https://github.com/scipy/scipy_doctest#tailoring-your-doctesting-experience) section.
```
dt_config.rndm_markers = {'# unintialized'}

3. **Run Doctests**
dt_config.stopwords = {'plt.', 'plt.hist', 'plt.show'}

Doctesting is configured to execute on SciPy using the `dev.py` module.
dt_config.local_resources = {
'scipy_doctest.tests.local_file_cases.local_files': ['scipy_doctest/tests/local_file.txt'],
'scipy_doctest.tests.local_file_cases.sio': ['scipy_doctest/tests/octave_a.mat']
}

To run all doctests, use the following command:
```bash
python dev.py smoke-docs
dt_config.skiplist = {
'scipy.special.sinc',
'scipy.misc.who',
'scipy.optimize.show_options'
}
```

To run doctests on specific SciPy modules, e.g: `cluster`, use the following command:
If you don't set these attributes, the [default settings](https://github.com/scipy/scipy_doctest/blob/58ff06a837b7bff1dbac6560013fc6fd07952ae2/scipy_doctest/impl.py#L94) of the attributes are used.

```bash
python dev.py smoke-docs -s cluster
```

### Running Doctests on Other Packages/Projects
#### Alternative Checkers

If you want to run doctests on packages or projects other than SciPy, follow these steps:
By default, we use the floating-point aware `DTChecker`. If you want to use an
alternative checker, all you need to do is to define the corresponding class,
and add an attribute to the `DTConfig` instance. For example,

1. **Install the plugin**

```bash
pip install scipy-doctest
```
class VanillaOutputChecker(doctest.OutputChecker):
"""doctest.OutputChecker to drop in for DTChecker.

LSP break: OutputChecker does not have __init__,
here we add it to agree with DTChecker.
"""
def __init__(self, config):
pass
```

2. **Register or Load the Plugin**
and

Next, you need to register or load the pytest plugin within your test module or `conftest.py` file.
```
dt_config = DTConfig()
dt_config.CheckerKlass = VanillaOutputChecker
```

To do this, add the following line of code:
See [a pytest example](https://github.com/scipy/scipy_doctest/blob/main/scipy_doctest/tests/test_pytest_configuration.py#L63)
and [a doctest example](https://github.com/scipy/scipy_doctest/blob/main/scipy_doctest/tests/test_runner.py#L94)
for more details.

```python
# In your conftest.py file or test module

pytest_plugins = "scipy_doctest"
```
### The SciPy Doctest Pytest Plugin

Check out the [pytest documentation](https://docs.pytest.org/en/stable/how-to/writing_plugins.html#requiring-loading-plugins-in-a-test-module-or-conftest-file) for more information on requiring/loading plugins in a test module or `conftest.py` file.
The pytest plugin enables the use of `scipy_doctest` tools to perform doctests.

3. **Configure your doctesting experience**
Follow the given instructions to utilize the pytest plugin for doctesting.

An in-depth explanation is given in the [tailoring your doctesting experience](https://github.com/scipy/scipy_doctest#tailoring-your-doctesting-experience) section.
### NumPy and SciPy wrappers

4. **Run doctests**

Once the plugin is registered, you can run your doctests by executing the following command:
NumPy wraps `scipy-doctest` with the `spin` command

```bash
$ python -m pytest --doctest-modules
```
or
```bash
$ pytest --pyargs <your-package> --doctest-modules
$ spin check-docs
```

By default, all doctests are collected. To only collect public objects, `strategy="api"`,
use the command flag
SciPy wraps `scipy-doctest` with custom `dev.py` commands:

```bash
$ pytest --pyargs <your-package> --doctest-modules --doctest-collect=api
```

### Tailoring Your Doctesting Experience

[DTConfig](https://github.com/scipy/scipy_doctest/blob/main/scipy_doctest/impl.py#L23) offers a variety of attributes that allow you to fine-tune your doctesting experience.

These attributes include:
1. **default_namespace (dict):** Defines the namespace in which examples are executed.
2. **check_namespace (dict):** Specifies the namespace for conducting checks.
3. **rndm_markers (set):** Provides additional directives which act like `# doctest: + SKIP`.
4. **atol (float) and rtol (float):** Sets absolute and relative tolerances for validating doctest examples.
Specifically, it governs the check using `np.allclose(want, got, atol=atol, rtol=rtol)`.
5. **optionflags (int):** These are doctest option flags.
The default setting includes `NORMALIZE_WHITESPACE` | `ELLIPSIS` | `IGNORE_EXCEPTION_DETAIL`.
6. **stopwords (set):** If an example contains any of these stopwords, the output is not checked (though the source's validity is still assessed).
7. **pseudocode (list):** Lists strings that, when found in an example, result in no doctesting. This resembles the `# doctest +SKIP` directive and is useful for pseudocode blocks or similar cases.
8. **skiplist (set):** A list of names of objects with docstrings known to fail doctesting and are intentionally excluded from testing.
9. **user_context_mgr:** A context manager for running tests.
Typically, it is entered for each DocTest (especially in API documentation), ensuring proper testing isolation.
10. **local_resources (dict):** Specifies local files needed for specific tests. The format is `{test.name: list-of-files}`. File paths are relative to the path of `test.filename`.
11. **parse_namedtuples (bool):** Determines whether to perform a literal comparison (e.g., `TTestResult(pvalue=0.9, statistic=42)`) or extract and compare the tuple values (e.g., `(0.9, 42)`). The default is `True`.
12. **nameerror_after_exception (bool):** Controls whether subsequent examples in the same test, after one has failed, may raise spurious NameErrors. Set to `True` if you want to observe these errors or if your test is expected to raise NameErrors. The default is `False`.

To set any of these attributes, create an instance of `DTConfig` called `dt_config`.
This instance is already set as an [attribute of pytest's `Config` object](https://github.com/scipy/scipy_doctest/blob/58ff06a837b7bff1dbac6560013fc6fd07952ae2/scipy_doctest/plugin.py#L39).

**Example:**

```python
dt_config = DTConfig()
dt_config.stopwords = {'plt.', '.hist', '.show'}
dt_config.local_resources = {
'scipy_doctest.tests.local_file_cases.local_files': ['scipy_doctest/tests/local_file.txt'],
'scipy_doctest.tests.local_file_cases.sio': ['scipy_doctest/tests/octave_a.mat']
}
dt_config.skiplist = {
'scipy.special.sinc',
'scipy.misc.who',
'scipy.optimize.show_options'
}
$ python dev.py smoke-docs # check docstrings
$ python dev.py smoke-tutorials # ReST user guide tutorials
```

If you don't set these attributes, the [default settings](https://github.com/scipy/scipy_doctest/blob/58ff06a837b7bff1dbac6560013fc6fd07952ae2/scipy_doctest/impl.py#L94) of the attributes are used.

By following these steps, you will be able to effectively use the SciPy Doctest pytest plugin for doctests in your Python projects.

Happy testing!

## Rough edges and sharp bits

Expand Down Expand Up @@ -307,7 +328,7 @@ being optional. So you either guard the imports in doctests (yikes!), or
the collections fails if dependencies are not available.

The solution is to explicitly `--ignore` the paths to modules with optionals.
(or use `DTConfig.pytest_extra_ignore` list):
(or, equivalently, use `DTConfig.pytest_extra_ignore` list):

```
$ pytest --ignore=/build-install/lib/scipy/python3.10/site-packages/scipy/_lib ...
Expand Down Expand Up @@ -351,20 +372,17 @@ leads to
differences are: (i) `pytest-doctestplus` is more sensitive to formatting,
including whitespace---thus if numpy tweaks its output formatting, doctests
may start failing; (ii) there is still a need for `# doctest: +FLOAT_CMP`
directives; (iii) being a pytest plugin, `pytest-doctestplus` is tightly
coupled to `pytest`. It thus needs to follow `pytest` releases, and
some maintenance work may be required to adapt when `pytest` publishes a new
release.
directives.

This project takes a different approach: in addition to plugging into `pytest`,
we closely follow the `doctest` API and implementation, which are naturally
way more stable then `pytest`.

- `NumPy` and `SciPy` use modified doctesting in their `refguide-check` utilities.
- `NumPy` and `SciPy` were using modified doctesting in their `refguide-check` utilities.
These utilities are tightly coupled to their libraries, and have been reported
to be not easy to reason about, work with, and extend to other projects.

This project is nothing but the core functionality of the modified
This project is mainly the core functionality of the modified
`refguide-check` doctesting, extracted to a separate package.
We believe having it separate simplifies both addressing the needs of these
two packages, and potential adoption by other projects.
Expand Down
8 changes: 8 additions & 0 deletions scipy_doctest/impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,14 @@ class DTConfig:
adding `# may vary` to the outputs of all examples.
Each key is a doctest name to skip, and the corresponding value is
a string. If not empty, the string value is used as the skip reason.
CheckerKlass : object, optional
The class for the Checker object. Must mimic the ``DTChecker`` API:
subclass the `doctest.OutputChecker` and make the constructor signature
read ``__init__(self, config=None)``, where `config` is a ``DTConfig``
instance.
This class will be instantiated by ``DTRunner``.
Defaults to `DTChecker`.

"""
def __init__(self, *, # DTChecker configuration
CheckerKlass=None,
Expand Down