Partial implementation of a vulnerability-config-based worked example…

… using TUDelft flood data. (#349) * Generation of config via tests and tidy. Signed-off-by: Joe Moorhouse <[email protected]> * Generation of config via tests and tidy. Signed-off-by: Joe Moorhouse <[email protected]> * Simplify maintenance of requirements-docs.txt. Signed-off-by: Joe Moorhouse <[email protected]> * Tidy documentation; reorder for increased use of notebooks. Signed-off-by: Joe Moorhouse <[email protected]> * Lint and tidy. Signed-off-by: Joe Moorhouse <[email protected]> * Apply extra beta distribution check. Signed-off-by: Joe Moorhouse <[email protected]> --------- Signed-off-by: Joe Moorhouse <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
os-climate · Sep 15, 2024 · 059613c · 059613c
1 parent c907459
commit 059613c
Show file tree

Hide file tree

Showing 83 changed files with 3,768 additions and 1,346 deletions.
diff --git a/.gitignore b/.gitignore
@@ -45,6 +45,7 @@ htmlcov/
 nosetests.xml
 coverage.xml
 *.cover
+tests/working/*
 
 # Translations
 *.mo
@@ -96,6 +97,7 @@ target/
 src/test/data/coords.json
 credentials.env
 .pdm-python
+.python-version
 
 # Latex
 methodology/PhysicalRiskMethodology.aux

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -19,7 +19,7 @@ repos:
     rev: v1.5.5
     hooks:
       - id: remove-tabs
-        exclude: ^.*\.(tsv)$
+        exclude: ^.*(\.(tsv)|Makefile)$
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.5.7

diff --git a/docs/Makefile b/docs/Makefile
@@ -10,11 +10,11 @@ BUILDDIR      = _build
 
 # Put it first so that "make" without argument is like "make help".
 help:
-    @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
 .PHONY: help Makefile
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
 %: Makefile
-    @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/docs/notebooks/2_asset_level_impacts.ipynb → docs/examples/asset_level_impacts.ipynb b/docs/notebooks/2_asset_level_impacts.ipynb → docs/examples/asset_level_impacts.ipynb
diff --git a/docs/notebooks/1_hazard_indicators.ipynb → docs/examples/hazard_indicators.ipynb b/docs/notebooks/1_hazard_indicators.ipynb → docs/examples/hazard_indicators.ipynb
diff --git a/docs/getting-started.rst b/docs/getting-started.rst
@@ -4,5 +4,5 @@ Getting started
 .. toctree::
    :maxdepth: 1
 
-   notebooks/1_hazard_indicators
-   notebooks/2_asset_level_impacts
+   examples/hazard_indicators
+   examples/asset_level_impacts
diff --git a/docs/index.rst b/docs/index.rst
@@ -95,7 +95,7 @@ Contents
 ==========
 
 .. toctree::
-   :maxdepth: 2
+  :maxdepth: 2
 
   getting-started
   methodology

diff --git a/docs/requirements-docs.txt b/docs/requirements-docs.txt
@@ -1,27 +1,218 @@
-sphinx==5.3.0
-sphinxcontrib-details-directive
+# This file is @generated by PDM.
+# Please do not edit it manually.
+
+accessible-pygments==0.0.5
+affine==2.4.0
+aiobotocore==2.15.0
+aiohappyeyeballs==2.4.0
+aiohttp==3.10.5
+aioitertools==0.12.0
+aiosignal==1.3.1
+alabaster==0.7.16
+annotated-types==0.7.0
+anyio==4.4.0
+apeye==1.4.1
+apeye-core==1.1.5
+appdirs==1.4.4
+appnope==0.1.4; platform_system == "Darwin"
+asciitree==0.3.3
+asttokens==2.4.1
+async-timeout==4.0.3; python_version < "3.11"
+attrs==24.2.0
+autodocsumm==0.2.13
+babel==2.16.0
+beautifulsoup4==4.12.3
+bleach==6.1.0
+blinker==1.8.2
+botocore==1.35.16
+brotli==1.1.0; platform_python_implementation == "CPython"
+brotlicffi==1.1.0.0; platform_python_implementation != "CPython"
+cachecontrol[filecache]==0.14.0
+cachetools==5.5.0
+certifi==2024.8.30
+cffi==1.17.1
+cfgv==3.4.0
+chardet==5.2.0
+charset-normalizer==3.3.2
+click==8.1.7
+colorama==0.4.6
+comm==0.2.2
+coverage[toml]==7.6.1
+cssselect2==0.7.0
+cssutils==2.11.1
+debugpy==1.8.5
+decorator==5.1.1
+defusedxml==0.7.1
+dep-logic==0.4.6
+dependency-injector==4.42.0
+dict2css==0.3.0.post1
+distlib==0.3.8
+docutils==0.20.1
+domdf-python-tools==3.9.0
+exceptiongroup==1.2.2; python_version < "3.11"
+executing==2.1.0
+fasteners==0.19; sys_platform != "emscripten"
+fastjsonschema==2.20.0
+filelock==3.16.0
+findpython==0.6.1
+flexcache==0.3
+flexparser==0.3.1
+fonttools[woff]==4.53.1
+frozenlist==1.4.1
+fsspec==2024.9.0
+geopandas==1.0.1
+graphviz==0.20.3
+greenlet==3.1.0; (platform_machine == "win32" or platform_machine == "WIN32" or platform_machine == "AMD64" or platform_machine == "amd64" or platform_machine == "x86_64" or platform_machine == "ppc64le" or platform_machine == "aarch64") and python_version < "3.13"
+h11==0.14.0
+h3==3.7.7
+hishel==0.0.30
+html5lib==1.1
+httpcore==1.0.5
+httpx[socks]==0.27.2
+identify==2.6.0
+idna==3.9
+imagesize==1.4.1
+importlib-metadata==8.5.0
+iniconfig==2.0.0
+installer==0.7.0
+ipykernel==6.29.5
+ipython==8.18.1
+jedi==0.19.1
+jinja2==3.1.4
+jmespath==1.0.1
+jsonschema==4.23.0
+jsonschema-specifications==2023.12.1
+jupyter-cache==1.0.0
+jupyter-client==8.6.2
+jupyter-core==5.7.2
+jupyterlab-pygments==0.3.0
+latexcodec==3.0.0
+libsass==0.23.0
+llvmlite==0.43.0
+lmdb==1.5.1
+lmdbm==0.0.6
+markdown-it-py==3.0.0
+markupsafe==2.1.5
+matplotlib-inline==0.1.7
+mdit-py-plugins==0.4.2
+mdurl==0.1.2
+mistune==3.0.2
+more-itertools==10.5.0
+msgpack==1.1.0
+multidict==6.1.0
+mypy==1.11.2
+mypy-extensions==1.0.0
+myst-nb==1.1.1
+myst-parser==3.0.1
+natsort==8.4.0
+nbclient==0.10.0
+nbconvert==7.16.4
+nbformat==5.10.4
+nbsphinx==0.9.5
+nest-asyncio==1.6.0
+nodeenv==1.9.1
+numba==0.60.0
+numcodecs==0.12.1
+numpy==2.0.2
 numpydoc==1.8.0
-sphinx-copybutton==0.5.0
-myst-nb==0.16.0
-pydata-sphinx-theme==0.15.3
-sphinxcontrib-bibtex
-sphinx-toolbox
-sphinx_toggleprompt==0.2.0
-sphinx_design==0.3
-sphinx_rtd_theme
-sphinx-toolbox
-sphinx-simplepdf
-pandoc
-nbsphinx
-graphviz
-affine==2.3.0
-numpy==1.22.0
-pydantic==1.10.13
+packaging==24.1
+pandas==2.2.2
+pandoc==2.4
+pandocfilters==1.5.1
+parso==0.8.4
+pbs-installer==2024.9.9
+pdm[pytest]==2.18.2
+pexpect==4.9.0; sys_platform != "win32"
+pillow==10.4.0
+pint==0.24.3
+platformdirs==4.3.3
+pluggy==1.5.0
+plumbum==1.8.3
+ply==3.11
+pre-commit==3.8.0
+prompt-toolkit==3.0.47
+psutil==6.0.0
+ptyprocess==0.7.0; sys_platform != "win32"
+pure-eval==0.2.3
+pybtex==0.24.0
+pybtex-docutils==1.0.3
+pycparser==2.22
+pydantic==2.9.1
+pydantic-core==2.23.3
+pydata-sphinx-theme==0.15.4
+pydyf==0.11.0
+pygments==2.18.0
+pyogrio==0.9.0
+pyphen==0.16.0
+pyproj==3.6.1
+pyproject-api==1.7.1
+pyproject-hooks==1.1.0
+pytest==8.3.3
+pytest-cov==5.0.0
+pytest-mock==3.14.0
+python-dateutil==2.9.0.post0
 python-dotenv==1.0.1
+pytz==2024.2
+pywin32==306; platform_system == "Windows" and platform_python_implementation != "PyPy" or sys_platform == "win32" and platform_python_implementation != "PyPy"
+pyyaml==6.0.2
+pyzmq==26.2.0
+referencing==0.35.1
 requests==2.32.3
-scipy==1.11.1
-s3fs==2022.1.0
+resolvelib==1.0.1
+rich==13.8.1
+rpds-py==0.20.0
+ruamel-yaml==0.18.6
+ruamel-yaml-clib==0.2.8; platform_python_implementation == "CPython" and python_version < "3.13"
+ruff==0.6.5
+s3fs==2024.9.0
+scipy==1.13.1
+shapely==2.0.6
+shellingham==1.5.4
+six==1.16.0
+sniffio==1.3.1
+snowballstemmer==2.2.0
+socksio==1.0.0
+soupsieve==2.6
+sphinx==7.4.7
+sphinx-autodoc-typehints==2.3.0
+sphinx-copybutton==0.5.2
+sphinx-design==0.6.1
+sphinx-jinja2-compat==0.3.0
+sphinx-prompt==1.8.0
+sphinx-rtd-theme==2.0.0
+sphinx-simplepdf==1.6.0
+sphinx-tabs==3.4.5
+sphinx-toggleprompt==0.5.2
+sphinx-toolbox==3.8.0
+sphinxcontrib-applehelp==2.0.0
+sphinxcontrib-bibtex==2.6.3
+sphinxcontrib-details-directive==0.1.0
+sphinxcontrib-devhelp==2.0.0
+sphinxcontrib-htmlhelp==2.1.0
+sphinxcontrib-jquery==4.1
+sphinxcontrib-jsmath==1.0.1
+sphinxcontrib-qthelp==2.0.0
+sphinxcontrib-serializinghtml==2.0.0
+sqlalchemy==2.0.34
+stack-data==0.6.3
+tabulate==0.9.0
+tinycss2==1.3.0
+tomli==2.0.1; python_version < "3.11"
+tomlkit==0.13.2
+tornado==6.4.1
+tox==4.18.1
+traitlets==5.14.3
+truststore==0.9.2; python_version >= "3.10"
+typing-extensions==4.12.2
+tzdata==2024.1
+unearth==0.17.2
+urllib3==1.26.20
+virtualenv==20.26.4
+wcwidth==0.2.13
+weasyprint==62.3
+webencodings==0.5.1
+wrapt==1.16.0
+yarl==1.11.1
 zarr==2.18.2
-pillow==10.3.0
-dependency-injector==4.41.0
-numba==0.60.0
+zipp==3.20.2
+zopfli==0.2.3
diff --git a/docs/user-guide.rst b/docs/user-guide.rst
@@ -1,64 +1,13 @@
 User guide
-===============
+==========
 
 If you are looking for the methodology of Physrisk or information about the different hazard indicators and vulnerability models, please check the `methodology document <methodology.html>`_. If it is not library-related, it should be there. Please see also `GitHub Contributing <https://github.com/os-climate/physrisk/blob/main/CONTRIBUTING.md>`_ for how to get set up.
 
-This section documents the structures and conventions of Physrisk and assumes some familiarity with the calculation of Physical Climate Risk (see `methodology document <methodology.html>`_ introduction).
+The following sections document the structures and conventions of Physrisk and assumes some familiarity with the calculation of Physical Climate Risk (see `methodology document <methodology.html>`_ introduction).
 
-Introduction to Physrisk
-------------------------
-Physic comprises:
 
-* A :code:`HazardModel` that retrieves *hazard indicators* for different locations.
-* :code:`VulnerabilityModels` that assess the vulnerability of assets to different climate hazards. :code:`VulnerabilityModels` use hazard indicators requested from the :code:`HazardModel` to calculate the *impact* of a hazard on a collection of assets.
-* Financial models that use the impacts calculated by the :code:`VulnerabilityModels` to calculate risk measures and scores.
+.. toctree::
+   :maxdepth: 1
 
-:code:`VulnerabilityModels` request hazard indicators using an :code:`indicator_id` (e.g. 'flood_depth' for inundation, 'max_speed' for wind). It is the responsibility of the :code:`HazardModel` to select the source of the hazard indicator data. Although :code:`VulnerabilityModels` can request specific hazard indicators by using a :code:`HazardDataHint` (e.g. a flood depth data set generated by a particular organization), this is generally discouraged: the idea is that a vulnerability model works with any data set. Note that units of the quantity are provided to the :code:`VulnerabilityModel` by the :code:`HazardModel`.
-
-Hazard indicator data sets
--------------------------
-The :code:`HazardModel` retrieves hazard indicators in a number of ways and can be made composite in order to combine different ways of accessing the data. At time of writing the common cases are that:
-
-1. Hazard indicator data is stored in `Zarr <https://zarr.readthedocs.io/en/stable/>`_ format (in an arbitrary Zarr store, although S3 is a popular choice).
-2. Hazard indicator data is retrieved via call to an external API. This is mainly used when combining commercial data to the public-domain.
-
-In case 1, hazard indicators are stored as three dimensional arrays. The array is ordered :math:`(z, y, x)` where :math:`y` is the spatial :math:`y` coordinate, :math:`x` is the spatial :math:`x` coordinate and :math:`z` is an *index* coordinate. The *index* takes on different meanings according to the type of data being stored.
-
-Indicators can be either:
-
-* Acute (A): the data comprises a set of hazard intensities for different return periods. In this case *index* refers to the different return periods.
-* Parametric (P): the data comprises a set of parameters. Here *index* refers to the different parameters. The parameters may be single values, or *index* might refer to a set of thresholds. Parametric indicators are used for chronic hazards.
-
-As mentioned above, a :code:`VulnerabilityModel` only specifies the identifier of the hazard indicator that is required, as well as the climate scenario ID and the year of the future projection (in the common case where hints are not used). This means that hazard indicator ID uniquely defines the data. For example, a vulnerability model requesting 'flood depth' could have data returned from a variety of data sets, depending on how the :code:`HazardModel` is configured.
-
-In case of stored-down (or 'pre-generated) hazard indicator data, as opposed to data source via an API, configuring which data sets to use is done by defining a :code:`SourcePath`. This is a function that provides the path, a unique identifier, that provides the data set to be used based on the hazard indicator ID, scenario and year. A convenience class :code:`CoreInventorySourcePaths` creates :code:`SourcePath` instances based on a set of rules called 'selectors'. A selector is a rule such as 'use the flood data set from this particular institution'. 
-
-+-----------------------+-------------------------------+---------------------------------------+
-| Hazard class          | Indicator ID (type)           | Description                           |
-+=======================+===============================+=======================================+
-| CoastalInundation,    | flood_depth (A)               | Flood depth (m) for available         |
-| PluvialInundation,    |                               | return periods. This is unprotected   |
-| RiverineInundation    |                               | depth.                                |
-|                       +-------------------------------+---------------------------------------+
-|                       | sop (P)                       | Standard of protection                |
-|                       |                               | (as return period in years).          |
-+-----------------------+-------------------------------+---------------------------------------+
-| Fire                  | fire_probability (P)          | Annual probability that location      |
-|                       |                               | is in a wildfire zone.                |
-+-----------------------+-------------------------------+---------------------------------------+
-| Heat                  | mean_degree_days/above/index  | Mean mean-temperature degree days per |
-|                       | (P)                           | year above a set of temperature       |
-|                       |                               | threshold indices.                    |
-+-----------------------+-------------------------------+---------------------------------------+
-| Drought               | months/spei/12m/below/index   | Mean months per year where the 12     |
-|                       | (P)                           | month SPEI index is below a set of    |
-|                       |                               | indices.                              |
-+-----------------------+-------------------------------+---------------------------------------+
-| Wind                  | max_speed                     | Maximum 1 minute sustained wind speed |
-|                       | (A)                           | for available return periods.         |
-+-----------------------+-------------------------------+---------------------------------------+
-
-
-Event based modelling
----------------------
-The most common use case for physrisk at time of writing is to perform analyses of portfolios but with asset impacts treated separately or with heuristics defining the dependence of asset impacts. In such cases it is possible, and indeed efficient, to derive the marginal probability distribution of the impact of the asset, the :code:`ImpactDistrib`. However to capture more realistic dependence of impacts, event-based modelling is needed. Here the :code:`HazardModel` additionally supplies an array of simulated hazard indicator values to the :code:`VulnerabilityModel` for each asset location, which in turn samples an array of impacts from the vulnerability function. At time of writing, the event-based functionality is not merged into main. Although specific types of :code:`HazardModel` and :code:`VulnerabilityModel` are needed for the event-based case, these do not replace the existing calculation which is optimized. 
+   user-guide/introduction
+   user-guide/vulnerability_config