Documentation improvements (#136)

* sweep rules update * sphinx upper bound * aeon version * revert * docs * sweep revert * user guide start * install * images * images config * new deps * datasets example * pretty sure this is in aeon * api * remove from toctree * datasets notebook * remove wip examples * results format notebook * results format notebook run * evaluation notebook * fixes * fixes * install note
time-series-machine-learning · Dec 1, 2023 · c9df99d · c9df99d
1 parent c32c898
commit c9df99d
Show file tree

Hide file tree

Showing 262 changed files with 947 additions and 40,478 deletions.
diff --git a/.gitignore b/.gitignore
@@ -126,6 +126,7 @@ dmypy.json
 
 # Test output
 test_output/
+examples/generated_results/
 
 # Test datasts
 test_datasets/

diff --git a/README.md b/README.md
@@ -4,29 +4,29 @@
 [![docs-main](https://img.shields.io/readthedocs/tsml-eval/latest?logo=readthedocs&label=docs%20%28latest%29)](https://tsml-eval.readthedocs.io/en/latest/)
 [![codecov](https://img.shields.io/codecov/c/github/time-series-machine-learning/tsml-eval?label=codecov&logo=codecov)](https://codecov.io/gh/time-series-machine-learning/tsml-eval)
 [![pypi](https://img.shields.io/pypi/v/tsml-eval?logo=pypi&color=blue)](https://pypi.org/project/tsml-eval/)
+[![!conda](https://img.shields.io/conda/vn/conda-forge/tsml-eval?logo=anaconda&color=blue)](https://anaconda.org/conda-forge/tsml-eval)
 [![python-versions](https://img.shields.io/pypi/pyversions/tsml-eval?logo=python)](https://www.python.org/)
 [![black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![license](https://img.shields.io/badge/license-BSD%203--Clause-green?logo=style)](https://github.com/time-series-machine-learning/tsml-eval/blob/main/LICENSE)
 
 # tsml-eval
 
-tsml-eval contains benchmarking and evaluation tools for time series machine learning
+`tsml-eval` contains benchmarking and evaluation tools for time series machine learning
 algorithms.
 
-The current release of tsml-eval is v0.1.1.
+The current release of `tsml-eval` is v0.1.1.
 
-Installation
-------------
+## Installation
 
-tsml-eval is available on PyPI and can be installed via pip:
+`tsml-eval` is available on PyPI and can be installed via pip:
 
-    pip install tsml-eval
+```console
+pip install tsml-eval
+```
 
-More information available on out documentation webpage:
-https://tsml-eval.readthedocs.io/en/stable/
+More information available on our [documentation](https://tsml-eval.readthedocs.io/en/stable/installation.html).
 
-Acknowledgements
-----------------
+## Acknowledgements
 
 This work is supported by the UK Engineering and Physical Sciences Research Council
-(EPSRC) EP/W030756/1
+(EPSRC) EP/W030756/2
diff --git a/docs/_static/css/custom.css b/docs/_static/css/custom.css
@@ -0,0 +1,16 @@
+/* card images */
+
+.tsml-card-image-m {
+  width: 100%;
+  height: 150px;
+  object-fit: contain;
+  object-position: center;
+}
+
+.tsml-card-image,
+.tsml-card-image-l {
+  width: 100%;
+  height: 200px;
+  object-fit: contain;
+  object-position: center;
+}
diff --git a/docs/api.md b/docs/api.md
@@ -13,8 +13,29 @@ Functions for evaluating the performance of a model.
     :toctree: auto_generated/
     :template: function.rst
 
-    evaluation.metrics.clustering_accuracy
-    evaluation.metrics.davies_bouldin_score_from_file
+    evaluation.evaluate_classifiers
+    evaluation.evaluate_classifiers_from_file
+    evaluation.evaluate_classifiers_by_problem
+    evaluation.evaluate_clusterers
+    evaluation.evaluate_clusterers_from_file
+    evaluation.evaluate_clusterers_by_problem
+    evaluation.evaluate_regressors
+    evaluation.evaluate_regressors_from_file
+    evaluation.evaluate_regressors_by_problem
+    evaluation.evaluate_forecasters
+    evaluation.evaluate_forecasters_from_file
+    evaluation.evaluate_forecasters_by_problem
+    evaluation.storage.ClassifierResults
+    evaluation.storage.ClustererResults
+    evaluation.storage.ForecasterResults
+    evaluation.storage.RegressorResults
+    evaluation.storage.load_classifier_results
+    evaluation.storage.load_clusterer_results
+    evaluation.storage.load_forecaster_results
+    evaluation.storage.load_regressor_results
+    evaluation.efficiency_benchmark.compare_estimators
+    evaluation.efficiency_benchmark.benchmark_estimator
+    evaluation.metrics.clustering_accuracy_score
 ```
 
 ## Experiments: [tsml_eval.experiments](https://github.com/time-series-machine-learning/tsml-eval/tree/main/tsml_eval/experiments)
@@ -45,15 +66,25 @@ Public utility functions used elsewhere in the package.
     :toctree: auto_generated/
     :template: function.rst
 
-    utils.functions.str_in_nested_list
+    utils.arguments.parse_args
     utils.experiments.resample_data
     utils.experiments.stratified_resample_data
     utils.experiments.write_classification_results
     utils.experiments.write_regression_results
     utils.experiments.write_clustering_results
     utils.experiments.write_results_to_tsml_format
-    utils.experiments.validate_results_file
     utils.experiments.fix_broken_second_line
     utils.experiments.compare_result_file_resample
     utils.experiments.assign_gpu
+    utils.experiments.timing_benchmark
+    utils.functions.str_in_nested_list
+    utils.functions.pair_list_to_dict
+    utils.functions.time_to_milliseconds
+    utils.functions.rank_array
+    utils.memory_recorder.record_max_memory
+    utils.validation.is_sklearn_estimator
+    utils.validation.is_sklearn_classifier
+    utils.validation.is_sklearn_regressor
+    utils.validation.is_sklearn_clusterer
+    utils.validation.validate_results_file
 ```
diff --git a/docs/conf.py b/docs/conf.py
@@ -48,11 +48,14 @@
     "sphinx.ext.autosectionlabel",
     "sphinx.ext.intersphinx",
     "sphinx.ext.linkcode",
+    "sphinxext.opengraph",
     "numpydoc",
     "nbsphinx",
     "sphinx_design",
     "sphinx_issues",
     "sphinx_copybutton",
+    "sphinx_remove_toctrees",
+    "versionwarning.extension",
     "myst_parser",
 ]
 
@@ -92,9 +95,15 @@
 # Link to GitHub repo for github_issues extension
 issues_github_path = "time-series-machine-learning/tsml-eval"
 
-# sphinx-copybutton configuration
+# sphinx-copybutton
+
 copybutton_exclude = ".linenos, .gp, .go"
 
+# sphinx-remove-toctrees configuration
+# see https://github.com/pradyunsg/furo/pull/674
+
+remove_from_toctrees = ["auto_generated/*"]
+
 # nbsphinx
 
 nbsphinx_execute = "never"
@@ -171,3 +180,4 @@ def find_source():
 
 html_theme = "furo"
 html_static_path = ["_static"]
+html_css_files = ["css/custom.css"]
diff --git a/examples/images/wip.png → docs/images/wip.png b/examples/images/wip.png → docs/images/wip.png
diff --git a/docs/index.md b/docs/index.md
@@ -6,7 +6,7 @@ hide-toc: true
 
 <br>
 
-``tsml-eval`` is a package containing tools for running experiments with time series
+`tsml-eval` is a package containing tools for running experiments with time series
 machine learning algorithms and evaluating the results. The package currently supports
 the classification, regression and clustering algorithms.
 
@@ -22,7 +22,7 @@ Installation
 
 ^^^
 
-Instructions to install ``tsml-eval``.
+Instructions to install `tsml-eval`.
 
 +++
 
@@ -43,7 +43,7 @@ User Guide
 
 ^^^
 
-Guides for users for running experiments using ``tsml-eval``.
+Guides for users for running experiments or evaluation using `tsml-eval`.
 
 +++
 
@@ -57,55 +57,57 @@ User Guide
 
 :::
 
+
 :::{grid-item-card}
 :text-align: center
 
-Publications
+API
 
 ^^^
 
-Supplementary material for publications using ``tsml-eval``.
+``tsml-eval`` API documentation.
 
 +++
 
-```{button-ref} publications
+```{button-ref} api
 :color: primary
 :click-parent:
 :expand:
 
-Publications
+API
 ```
 
 :::
 
 :::{grid-item-card}
 :text-align: center
 
-API
+Publications
 
 ^^^
 
-``tsml-eval`` API documentation.
+Supplementary material for publications using that use `tsml-eval`.
 
 +++
 
-```{button-ref} api
+```{button-ref} publications
 :color: primary
 :click-parent:
 :expand:
 
-API
+Publications
 ```
 
 :::
 
 ::::
 
+
 ```{toctree}
 :hidden:
 
 installation.md
 user_guide.md
-publications.md
 api.md
+publications.md
 ```
diff --git a/docs/installation.md b/docs/installation.md
@@ -1,72 +1,110 @@
 # Installation
 
+The following contains information on installing `tsml-eval` for users and developers
+with write access. Those who wish to contribute to `tsml-eval` without write access
+will need to create a fork, see the [aeon](https://www.aeon-toolkit.org/en/stable/developer_guide/dev_installation.html)
+and [sklearn](https://scikit-learn.org/stable/developers/contributing.html#how-to-contribute)
+documentation on contributing and developer installation for guidance.
+
+We recommend setting up a fresh virtual environment or the conda equivalent before
+installing `tsml-eval`. See the [aeon guide](https://www.aeon-toolkit.org/en/stable/installation.html#using-a-pip-venv)
+for setup information.
+
 ## Install from PyPi
 
-The easiest way to install ``tsml-eval`` is using ``pip``:
+The easiest way to install `tsml-eval` is using `pip`:
 
-```{code-block} console
+```console
 pip install tsml-eval
 ```
 
 Some estimators require additional dependencies. You can install these individually as
-required, or install all of them using the ``all_extras`` extra dependency set:
+required, or install all of them using the `all_extras` extra dependency set:
 
-```{code-block} console
+```console
 pip install tsml-eval[all_extras]
 ```
 
+```{note}
+    If this results in a "no matches found" error, it may be due to how your shell
+    handles special characters. Try surrounding the dependency portion with quotes i.e.
+
+    pip install tsml-eval"[all_extras]"
+```
+
 All extra dependency sets can be found in the [pyproject.toml](https://github.com/time-series-machine-learning/tsml-eval/blob/main/pyproject.toml)
-file ``[project.optional-dependencies]`` options.
+file `[project.optional-dependencies]` options.
 
 To install a specific [release](https://github.com/time-series-machine-learning/tsml-eval/releases)
 version, specify the version number when installing:
 
-```{code-block} console
+```console
 pip install tsml-eval==0.1.0
 ```
 
-```{code-block} console
+```console
 pip install tsml-eval[all_extras]==0.1.0
 ```
 
+## Install from conda-forge
+
+`tsml-eval` is also available on [conda-forge](https://anaconda.org/conda-forge/tsml-eval).
+
+```console
+conda create -n tsml-env -c conda-forge tsml-eval
+conda activate tsml-env
+```
+
+Currently for conda installations, optional dependencies must be installed separately.
+
 ## Install fixed dependency versions for a publication
 
-``tsml-eval`` [publications](publications.md) contain a ``requirements.txt`` file that
-lists the versions of all  dependencies used to generate results at the time of the
-release.
+`tsml-eval` [publications](publications.md) contain a `static_publication_reqs.txt`
+file that lists the versions of all dependencies used to generate results at the time
+of the release.
 
 To install the dependencies using this file, run:
 
-```{code-block} console
-pip install -r requirements.txt
+```console
+pip install -r static_publication_reqs.txt
 ```
 
-## Install latest in-development version from GitHub
+## Install the latest in-development version from GitHub
 
-The latest development version of ``tsml-eval`` can be installed directly from GitHub
-using ``pip``:
+The latest development version of `tsml-eval` can be installed directly from GitHub
+using `pip`:
 
-```{code-block} console
+```console
 pip install git+https://github.com/time-series-machine-learning/tsml-eval.git@main
 ```
 
-## Install for developers
+The latest development version of dependencies can be installed this way, i.e. for
+`aeon`
+
+```console
+pip install git+https://github.com/aeon-toolkit/aeon.git@main
+```
+
+If you have a different version of `tsml-eval` or a dependency installed, you must
+uninstall it first before installing the development version.
+
+## Install for developers with write access
 
-To install ``tsml-eval`` for development, first clone the GitHub repository:
+To install `tsml-eval` for development, first clone the GitHub repository:
 
-```{code-block} console
+```console
 git clone https://github.com/time-series-machine-learning/tsml-eval.git
 ```
 
 Then install the package in editable mode with developer dependencies:
 
-```{code-block} console
+```console
 pip install --editable .[dev]
 ```
 
 We recommend setting up pre-commit hooks to automatically format code and check for
 common issues before committing:
 
-```{code-block} console
+```console
 pre-commit install
 ```