Skip to content

Commit

Permalink
docs close enough
Browse files Browse the repository at this point in the history
  • Loading branch information
jonschwenk committed Dec 21, 2024
1 parent 492519f commit e8006a3
Show file tree
Hide file tree
Showing 16 changed files with 76 additions and 55 deletions.
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -155,5 +155,5 @@ results/*.tif
results/*.csv
results/*.shx
data
hydropop/config.py
ecopop/config.py
config.py
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,5 @@ rabpro:

initial_data: data/initial_basins.gpkg data/initial_gages.gpkg

data/initial_basins.gpkg data/initial_gages.gpkg: hydropop/streamflow/selecting_gages.py
data/initial_basins.gpkg data/initial_gages.gpkg: ecopop/streamflow/selecting_gages.py
python $<
2 changes: 1 addition & 1 deletion docs/doctables/auxiliary_files.csv
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ directory,filename,description
watersheds,{name}_\ `basins.gpkg`_,watershed geometries and many attributes for which streamflow data exist; see detailed file description
watersheds,{name}_\ `gages.gpkg`_,streamgage locations and many gage attributes for which streamflow data exist; see detailed file description
streamflow,{id_gage_n}.csv,daily time series of streamflow in cubic meters per second; contact devs for information about q_quality column if desired
forcings,{hpu_id_n}.csv,time series of ERA5-Land-derived variables for each hydropop unit; see detailed file description
forcings,{hpu_id_n}.csv,time series of ERA5-Land-derived variables for each ecopop unit; see detailed file description
10 changes: 5 additions & 5 deletions docs/doctables/hp_output_files.csv
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
filename,description
{name}_\ `hpus.gpkg`_,GeoPackage containing the HPUs as polygons (or MultiPolygons) and some attributes. See the hpus.gpkg table for attribute list.
{name}_hpus.shp,ESRI shapefile of HPU boundaries; this is primarily used for upload to Google Earth Engine (GEE) as GEE does not support the .gpkg format.
{name}_hpus.tif,Geotiff of hydropop units. Pixel values represent the unit to which the HPU belongs and correspond to {name}_hpus.gpkg and {name}_hpus.shp. Provided as a convenience when raster math is faster.
{name}_hpu_classes.tif,"Geotiff of hydropop classes. Pixel values represent the class to which the HPU belongs. A HP class is the ""cluster"" to which the pixel belongs. The maximum number of classes is set as a hydropop creation parameter. This file is essentially a rasterization of {name}_hpu_classes.gpkg."
{name}_hpu_classes.gpkg,GeoPackage of hydropop classes. Each polygon represents a connected cluster of pixels sharing the same class. The maximum number of classes is set as a hydropop creation parameter. This is a polygonization of the {name}_hpu_classes.tif file.
{name}_areagrid.tif,"Geotiff for which pixel values represent the area of the pixel in square km. This is used for computing actual hydropop unit areas, as working in unprojected coordinate systems (4326) require a bit of extra work to estimate pixel areas in meaningful units (i.e. km instead of degrees)."
{name}_adjacency.csv,"Provides connectivity information among hydropop units. Contains two columns: **hpu_id** of each hydropop unit in the dataset, and **adjacency** that specifies each hpu_id's neighbors as a comma-separated string."
{name}_hpus.tif,Geotiff of ecopop units. Pixel values represent the unit to which the HPU belongs and correspond to {name}_hpus.gpkg and {name}_hpus.shp. Provided as a convenience when raster math is faster.
{name}_hpu_classes.tif,"Geotiff of ecopop classes. Pixel values represent the class to which the HPU belongs. A HP class is the ""cluster"" to which the pixel belongs. The maximum number of classes is set as a ecopop creation parameter. This file is essentially a rasterization of {name}_hpu_classes.gpkg."
{name}_hpu_classes.gpkg,GeoPackage of ecopop classes. Each polygon represents a connected cluster of pixels sharing the same class. The maximum number of classes is set as a ecopop creation parameter. This is a polygonization of the {name}_hpu_classes.tif file.
{name}_areagrid.tif,"Geotiff for which pixel values represent the area of the pixel in square km. This is used for computing actual ecopop unit areas, as working in unprojected coordinate systems (4326) require a bit of extra work to estimate pixel areas in meaningful units (i.e. km instead of degrees)."
{name}_adjacency.csv,"Provides connectivity information among ecopop units. Contains two columns: **hpu_id** of each ecopop unit in the dataset, and **adjacency** that specifies each hpu_id's neighbors as a comma-separated string."
2 changes: 1 addition & 1 deletion docs/doctables/hpu_gpkg.csv
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
attribute,units,description,source
fid,,unique feature id created upon export; essentially meaningless.,derived
hpu_id,,unique hydropop id,derived
hpu_id,,unique ecopop id,derived
pop_mean,n people per 0.01 km^2,average population density across the HU,"Worldpop Estimated Residential Population per 100x100m Grid Square for 2020, downloaded from `GEE <https://developers.google.com/earth-engine/datasets/catalog/WorldPop_GP_100m_pop>`_ and generated by `WorldPop <https://www.worldpop.org/>`_."
area_sum,km^2,area of the HU,derived
hpu_class,,class to which this HU belongs,derived
Expand Down
8 changes: 4 additions & 4 deletions docs/paper_fig/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -3,25 +3,25 @@
all: data figures

data:
python hydropop/dev/end_to_end_new.py \
python ecopop/dev/end_to_end_new.py \
--pop_breaks -11 -10 -4 0 100 \
--hthi_breaks -0.01 0.4 0.7 1.01 \
--path_bounding_box data/roi_small.gpkg \
--run_name coarse_coarse_small

python hydropop/dev/end_to_end_new.py \
python ecopop/dev/end_to_end_new.py \
--pop_breaks -11 -10 -4 0 100 \
--hthi_breaks -0.01 0.3 0.55 0.75 0.9 1.01 \
--path_bounding_box data/roi_small.gpkg \
--run_name coarse_fine_small

python hydropop/dev/end_to_end_new.py \
python ecopop/dev/end_to_end_new.py \
--pop_breaks -11 -10 -4 -1 1 2 100 \
--hthi_breaks -0.01 0.4 0.7 1.01 \
--path_bounding_box data/roi_small.gpkg \
--run_name fine_coarse_small

python hydropop/dev/end_to_end_new.py \
python ecopop/dev/end_to_end_new.py \
--pop_breaks -11 -10 -4 -1 1 2 100 \
--hthi_breaks -0.01 0.3 0.55 0.75 0.9 1.01 \
--path_bounding_box data/roi_small.gpkg \
Expand Down
2 changes: 1 addition & 1 deletion docs/source/_build/html/genindex.html
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul>
<li class="toctree-l1"><a class="reference internal" href="pages/installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="pages/hpdata.html">Hydropop Data</a></li>
<li class="toctree-l1"><a class="reference internal" href="pages/hpdata.html">Ecopop Data</a></li>
</ul>

</div>
Expand Down
4 changes: 2 additions & 2 deletions docs/source/_build/html/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
</div><div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="Navigation menu">
<ul>
<li class="toctree-l1"><a class="reference internal" href="pages/installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="pages/hpdata.html">Hydropop Data</a></li>
<li class="toctree-l1"><a class="reference internal" href="pages/hpdata.html">Ecopop Data</a></li>
</ul>

</div>
Expand Down Expand Up @@ -74,7 +74,7 @@ <h1>Welcome to hpdocs’s documentation!<a class="headerlink" href="#welcome-to-
<div class="toctree-wrapper compound">
<ul>
<li class="toctree-l1"><a class="reference internal" href="pages/installation.html">Installation</a></li>
<li class="toctree-l1"><a class="reference internal" href="pages/hpdata.html">Hydropop Data</a></li>
<li class="toctree-l1"><a class="reference internal" href="pages/hpdata.html">Ecopop Data</a></li>
</ul>
</div>
</section>
Expand Down
8 changes: 4 additions & 4 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
# -- Project information -----------------------------------------------------
# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information

project = "hpdocs"
project = "epdocs"
copyright = "2023, Jon Schwenk"
author = "Jon Schwenk"
release = "0.1"
Expand All @@ -15,9 +15,9 @@
# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
import os
import sys
# import hydropop
sys.path.insert(0, os.path.abspath(os.path.join("..", "..")))
# sys.path.insert(0, os.path.abspath("../"))
sys.path.insert(0, os.path.abspath('../'))
sys.path.append(r'C:\Users\318596\Desktop\ecopop\ecopop')
import ecopop

extensions = ["sphinx.ext.autodoc", "sphinx.ext.autosummary"]

Expand Down
21 changes: 21 additions & 0 deletions docs/source/pages/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,29 @@
API
===

ep_class
----------------------------

.. automodule:: ecopop.ep_class
:members:


ep_utils
----------------------------

.. automodule:: ecopop.ep_utils
:members:

.. API Reference
.. =============
.. .. automodule:: ecopop.ep_class
.. :members:
.. :undoc-members:
.. :show-inheritance:
.. .. automodule:: ecopop.ep_utils
.. :members:
.. :undoc-members:
.. :show-inheritance:
30 changes: 15 additions & 15 deletions docs/source/pages/hpdata.rst
Original file line number Diff line number Diff line change
@@ -1,29 +1,29 @@
.. _hpdata:
.. _epdata:
.. role:: raw-html(raw)
:format: html

=============
Hydropop Data
Ecopop Data
=============

A full "run" of hydropop creation results in a set of files that provide hydropop delineations and a suite of attributes and parameters we have identified as relevant to the E3SM-Land Model. This documentation outlines the structure and contents of these data. Note that additional data may be provided (for example, data related to streamflow gages and watersheds). Information about those tables are also provided here, but are not necessarily part of the core hydropop functionality.
A full "run" of ecopop creation results in a set of files that provide ecopop delineations and a suite of attributes and parameters we have identified as relevant to the E3SM-Land Model. This documentation outlines the structure and contents of these data. Note that additional data may be provided (for example, data related to streamflow gages and watersheds). Information about those tables are also provided here, but are not necessarily part of the core ecopop functionality.

Each "run" must be named, and that name is shown as {name} in the following documentation. Some filenames depend on the ids they represent, and these are shown in {} in the tree below. When the number of files in a directory can be variable, `...` is used.

Directory and file structure
----------------------------

The following tree shows how hydropop exports are structured on disk. Note that all core files are contained in the parent directory (called ``{name}``), while auxiliary/optional files will be in subdirectories that may or may not be present depending on the run parameters. That is, all files in the ``{name}`` directory will always exist for any hydropop export, but other directories (e.g. ``streamflow``, ``watersheds``, ``forcings``, etc.) might not.
The following tree shows how ecopop exports are structured on disk. Note that all core files are contained in the parent directory (called ``{name}``), while auxiliary/optional files will be in subdirectories that may or may not be present depending on the run parameters. That is, all files in the ``{name}`` directory will always exist for any ecopop export, but other directories (e.g. ``streamflow``, ``watersheds``, ``forcings``, etc.) might not.


::

{name} <-- parent directory
├── {name}_hpus.gpkg
├── {name}_hpus.tif
├── {name}_hpus.shp
├── {name}_hpu_classes.gpkg
├── {name}_hpu_classes.tif
├── {name}_epus.gpkg
├── {name}_epus.tif
├── {name}_epus.shp
├── {name}_epu_classes.gpkg
├── {name}_epu_classes.tif
├── {name}_areagrid.tif
├── {name}_adjacency.csv
├── LICENCE.txt
Expand All @@ -33,7 +33,7 @@ The following tree shows how hydropop exports are structured on disk. Note that
└── {id_gage_N}.csv
├── watersheds
├── {name}_basins.gpkg
├── {name}_hpu_gages.csv
├── {name}_epu_gages.csv
└── {name}_gages.gpkg
├── forcings
├── daily
Expand All @@ -49,7 +49,7 @@ The following tree shows how hydropop exports are structured on disk. Note that
Core files
----------

The following files will be present for any hydropop exports. Note that hydropop processing is all done in unprojected coordinates (EPSG:4326), but care is taken to correctly compute areas and distances when appropriate. All georeferenced outputs therefore also are in the EPSG:4326 coordinate reference system.
The following files will be present for any ecopop exports. Note that ecopop processing is all done in unprojected coordinates (EPSG:4326), but care is taken to correctly compute areas and distances when appropriate. All georeferenced outputs therefore also are in the EPSG:4326 coordinate reference system.

.. csv-table::
:file: ../../doctables/hp_output_files.csv
Expand All @@ -60,13 +60,13 @@ The following files will be present for any hydropop exports. Note that hydropop
Auxiliary files
---------------

Auxiliary files are not considered part of the core hydropop functionality, and therefore these files may not be present for a general user.
Auxiliary files are not considered part of the core ecopop functionality, and therefore these files may not be present for a general user.

The ``watersheds`` directory contains files about streamflow gages and their watersheds. These data were obtained from the Veins of the Earth (VotE) data platform, which is currently only available to LANL employees. Information about VotE can be found `here <https://www.essoar.org/doi/10.1002/essoar.10509913.2>`_, and LANL collaborators can access the private VotE repository upon request.

The ``streamflow`` directory contains one `csv` per streamflow gage, and each filename corresponds to the `id_gage` provided by VotE (and found within the `watersheds` files). These data were obtained from the Veins of the Earth (VotE) data platform, which is currently only available to LANL employees. Information about VotE can be found `here <https://www.essoar.org/doi/10.1002/essoar.10509913.2>`_, and LANL collaborators can access the private VotE repository upon request.

The ``forcings`` directory contains meterologic and other time-series data either required for or useful to running E3SM-Land models on hydropop units. The data were sampled from the `ERA5-Land Hourly <https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_LAND_HOURLY>`_ dataset on Google Earth Engine (GEE). Some postprocessing to bring units to more standard formats (and other details) is performed, so the band descriptions provided by the GEE Data Catalog might not be exactly accuarte. Each `csv` file is named a corresponding hydropop id and contains time series. See the `hpus.gpkg`_ for more detailed descriptions of the contents of each file.
The ``forcings`` directory contains meterologic and other time-series data either required for or useful to running E3SM-Land models on ecopop units. The data were sampled from the `ERA5-Land Hourly <https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_LAND_HOURLY>`_ dataset on Google Earth Engine (GEE). Some postprocessing to bring units to more standard formats (and other details) is performed, so the band descriptions provided by the GEE Data Catalog might not be exactly accuarte. Each `csv` file is named a corresponding ecopop id and contains time series. See the `hpus.gpkg`_ for more detailed descriptions of the contents of each file.

.. csv-table::
:file: ../../doctables/auxiliary_files.csv
Expand All @@ -77,7 +77,7 @@ The ``forcings`` directory contains meterologic and other time-series data eithe
Individual file contents
------------------------

_`hpus.gpkg`
_`epus.gpkg`
^^^^^^^^^^^^
.. csv-table::
:file: ../../doctables/hpu_gpkg.csv
Expand All @@ -94,7 +94,7 @@ _`basins.gpkg` and _`gages.gpkg`
:header-rows: 1


Forcings csvs aka _`{hpu_id_n}.csv`
Forcings csvs aka _`{epu_id_n}.csv`
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
.. csv-table::
:file: ../../doctables/forcings.csv
Expand Down
18 changes: 9 additions & 9 deletions docs/source/pages/installation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -19,28 +19,28 @@ It's a package-solver that's orders of magnitude faster. You can install it into

or follow the actual `installation instructions <https://mamba.readthedocs.io/en/latest/installation.html>`_.

Clone the ``hydropop`` repo to your PC. This will include the ``hp_enviornment.yml`` file containing the dependencies. I
Clone the ``ecopop`` repo to your PC. This will include the ``enviornment.yml`` file containing the dependencies. I
prefer `Github Desktop <https://desktop.github.com/>`_ for repo management and cloning.

Open an Anaconda Terminal window and create an empty environment named ``hprepo``:
Open an Anaconda Terminal window and create an empty environment named ``ecopop``:

::

conda create --name hprepo --no-default-packages
conda create --name ecopop --no-default-packages

Then update this enviornment using ``mamba`` and pointing to the ``hp_environment.yml`` file:
Then update this enviornment using ``mamba`` and pointing to the ``environment.yml`` file:

::

mamba env update -n hprepo --file "path\to\hp_environment.yml"
mamba env update -n ecopop --file "path\to\environment.yml"

Finally, use ``pip`` to do a live-install of the repo into the environment.

::

conda activate hprepo
pip install -e "path\to\hydropop\repo\folder"
conda activate ecopop
pip install -e "path\to\ecopop\repo\folder"

And that's it! Note that this is a "live install" of the hydropop repo, which means that as you edit (or update) the
repo's code on your PC, those changes will be immediately recognized. You should be able to use ``import hydropop.hp_utils``
And that's it! Note that this is a "live install" of the ecopop repo, which means that as you edit (or update) the
repo's code on your PC, those changes will be immediately recognized. You should be able to use ``import ecopop.hp_utils``
etc. at this point.
2 changes: 1 addition & 1 deletion docs/source/pages/terminology.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
:format: html

====================
Hydropop Terminology
Ecopop Terminology
====================

- hthi: hydrotopo hab index - a combination of four indicators of water availability likelihood (distance to persistent waterbody, topographic wetness, height above nearest drainage, human created habitat)
Expand Down
4 changes: 2 additions & 2 deletions ecopop/dev/determine_hthi_pop_ranges.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
from osgeo import gdal
import numpy as np
import sys
sys.path.append(r'X:\Research\CIMMID\hydropop\make_hpus')
sys.path.append(r'X:\Research\CIMMID\ecopop\make_hpus')
import hp_class as hpc
from matplotlib import pyplot as plt
import matplotlib.patches as mpatches
Expand Down Expand Up @@ -123,7 +123,7 @@
plt.imshow(Iplot, interpolation='none')
plt.legend(handles=patches, loc=0, borderaxespad=0., title='HTHI range')

# Plot hydropop units
# Plot ecopop units
lid = 1
labels = {}
Ic_hpu = np.zeros(test.I['hthi'].shape, dtype=int)
Expand Down
8 changes: 4 additions & 4 deletions ecopop/ep_class.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@
from scipy.cluster.vq import kmeans2
from skimage.graph import RAG
#
import ep_utils as eut
import rivgraph_ports as io
from ecopop import ep_utils as eut
from ecopop import rivgraph_ports as io


"""
Expand Down Expand Up @@ -90,14 +90,14 @@ def __init__(self, path_pop, path_hthi, path_water_raster=None, bounding=None, w

def compute_ep_classes_kmeans(self, n_groups):
"""
Creates an image where each pixel value is the HP group to which
Creates an image where each pixel value is the EP group to which
the pixel belongs. Pixels are grouped via a k-means clustering based
on the (population, hab index) for each pixel. The number of groups
must be specified and can be thought of as the number of regions
in which the population, hab index space is divided into.
n_groups is NOT the total number of ecopop units, but the number
of HP unit types.
of EP unit types.
self.centroids contains the centroid of the (population, hab index)
"coordinates" of each group--there is no spatial information here.
Expand Down
8 changes: 4 additions & 4 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,16 @@
long_description = fh.read()

setuptools.setup(
name="hydropop",
name="ecopop",
version="1.0",
author="Jon Schwenk",
author_email="[email protected]",
description="Package to create hydropop scaling units",
description="Package to create ecopop scaling units",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/lanl/hydropop",
url="https://github.com/lanl/ecopop",
project_urls={
"Bug Tracker": "https://github.com/lanl/hydropop/issues",
"Bug Tracker": "https://github.com/lanl/ecopop/issues",
},
# scripts=["rabpro/cli/rabpro"],
python_requires='>=3',
Expand Down

0 comments on commit e8006a3

Please sign in to comment.