From 93eb17a2c6b789d83c893ebc783b9f88402c9fef Mon Sep 17 00:00:00 2001 From: vincentsarago Date: Wed, 25 Oct 2023 18:56:24 +0200 Subject: [PATCH 1/2] refactor docs --- CONTRIBUTING.md | 18 +++++- README.md | 25 +++++--- docker-compose.official.yml | 10 ++-- docs/mkdocs.yml | 24 ++++---- docs/src/contributing.md | 21 +++++-- docs/src/customization.md | 67 +++++++++++++++++++++ docs/src/deployment.md | 24 ++++---- docs/src/intro.md | 33 +++++++---- docs/src/services.md | 113 +++++++++++++++++++++++++----------- 9 files changed, 249 insertions(+), 86 deletions(-) create mode 100644 docs/src/customization.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d1f93b2..96d5774 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,7 @@ -# Development - Contributing - Issues and pull requests are more than welcome: https://github.com/developmentseed/eoAPI/issues +You can also start **Discussions** in https://github.com/developmentseed/eoAPI/discussions + **dev install** ```bash @@ -19,7 +19,9 @@ python -m pip install "psycopg[binary,pool]" python -m pip install -e runtime/eoapi/raster["test"] # or -e runtime/eoapi/stac["test"] | -e runtime/eoapi/vector["test"] ``` -Note: services might have incompatible dependencies which you can resolve by using virtual environnement per service +!!! danger + + Python applications might have incompatible dependencies which you can resolve by using virtual environment *per application* **pre-commit** @@ -28,3 +30,13 @@ This repo is set to use `pre-commit` to run *isort*, *ruff*, *pydocstring*, *bla ```bash $ pre-commit install ``` + +### Open Source + +You can also contribute indirectly to eoAPI by helping on the sub-modules: + +- **PgSTAC** database https://github.com/stac-utils/pgstac +- **stac-fastapi**: https://github.com/stac-utils/stac-fastapi +- **titiler-pgstac**: https://github.com/stac-utils/titiler-pgstac +- **TiTiler**: https://github.com/developmentseed/titiler +- **TiPg**: https://github.com/developmentseed/tipg diff --git a/README.md b/README.md index ca3dc10..c376b72 100644 --- a/README.md +++ b/README.md @@ -44,11 +44,22 @@ ## Getting started -- Clone the repository: `git clone https://github.com/developmentseed/eoAPI.git` -- Navigate to the project: `cd eoAPI` -- Run services with `docker compose up` -- Follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](./demo/)) to load some data into eoAPI -- Check out the [Search Viewer](http://localhost:8081/index.html), and the API documentation ([STAC Metadata](http://localhost:8081/docs), [Raster Tiles](http://localhost:8082/docs), [Vector Tiles](http://localhost:8083/api.html)) +The easiest way to start exploring the different eoAPI services is with *Docker*. Clone this repository and start the multi-container *Docker* applications using `Compose`: + +``` +git clone https://github.com/developmentseed/eoAPI.git +cd eoAPI +docker compose up +``` + +Once the application are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). + +Then you can start exploring your dataset with: + + - the STAC Metadata service [http://localhost:8081](http://localhost:8081) + - the Raster service [http://localhost:8082](http://localhost:8082) + +If you've added vector dataset to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). Alternatively, you may launch the application locally: ```bash @@ -70,9 +81,9 @@ Note: services might have incompatible dependencies which you can resolve by usi ## Deployment -This repository has current runtimes that are consistently updated with new functionality. +This repository has current runtimes that are consistently updated with new functionality. -The services can be deployed locally via docker with `docker-compose up`. The official runtimes can be launched with `docker-compose -f docker-compose.yml -f docker-compose.official.yml up stac-fastapi titiler-pgstac tipg`. +The services can be deployed locally via docker with `docker-compose up`. The official runtimes can be launched with `docker compose -f docker-compose.official.yml up stac-fastapi titiler-pgstac tipg`. Two Infrastructure as Code (IaC) repositories are available: - [eoapi-cdk](https://github.com/developmentseed/eoapi-cdk): A set of AWS CDK constructs to deploy eoAPI services diff --git a/docker-compose.official.yml b/docker-compose.official.yml index 0aa56a7..b6f54c4 100644 --- a/docker-compose.official.yml +++ b/docker-compose.official.yml @@ -2,6 +2,9 @@ version: '3' services: stac-fastapi: + # Note: + # the official ghcr.io/stac-utils/stac-fastapi-pgstac image uses python 3.8 and uvicorn + # which is why here we use a custom Dockerfile using python 3.11 and gunicorn build: context: . dockerfile: dockerfiles/Dockerfile.stac @@ -40,7 +43,7 @@ services: # At the time of writing, rasterio and psycopg wheels are not available for arm64 arch # so we force the image to be built with linux/amd64 platform: linux/amd64 - image: ghcr.io/stac-utils/titiler-pgstac:latest + image: ghcr.io/stac-utils/titiler-pgstac:0.8.0 ports: - "${MY_DOCKER_IP:-127.0.0.1}:8082:8082" environment: @@ -86,7 +89,7 @@ services: - ./dockerfiles/scripts:/tmp/scripts tipg: - image: ghcr.io/developmentseed/tipg:latest + image: ghcr.io/developmentseed/tipg:0.4.4 ports: - "${MY_DOCKER_IP:-127.0.0.1}:8083:8083" environment: @@ -115,8 +118,7 @@ services: - ./dockerfiles/scripts:/tmp/scripts database: - container_name: eoapi.db - image: ghcr.io/stac-utils/pgstac:v0.7.10 + image: ghcr.io/stac-utils/pgstac:v0.8.1 environment: - POSTGRES_USER=username - POSTGRES_PASSWORD=password diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 3f17da6..b7d5738 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -19,12 +19,14 @@ extra: link: "https://developmentseed.org/blog" nav: - - Home: "index.md" - - Intro: "intro.md" - - Services: "services.md" - - Deployment: "deployment.md" - - Contributing: "contributing.md" - + - Home: index.md + - Getting started: + - intro.md + - Services: services.md + - Contributing: contributing.md + - Advanced User Guide: + - Customization: customization.md + - Deployment: deployment.md plugins: - search @@ -36,17 +38,17 @@ theme: palette: primary: custom accent: deep orange - features: - navigation.tabs + - navigation.tabs.sticky - navigation.instant - navigation.tracking - navigation.top - - navigation.expand + - navigation.footer + - navigation.sections favicon: img/favicon.ico logo: img/logo.png - markdown_extensions: - admonition - attr_list @@ -59,7 +61,6 @@ markdown_extensions: - pymdownx.caret: insert: false - pymdownx.details - - pymdownx.emoji - pymdownx.escapeall: hardbreak: true nbsp: true @@ -73,3 +74,6 @@ markdown_extensions: - pymdownx.tilde - toc: permalink: true + - pymdownx.emoji: + emoji_index: !!python/name:material.extensions.emoji.twemoji + emoji_generator: !!python/name:material.extensions.emoji.to_svg diff --git a/docs/src/contributing.md b/docs/src/contributing.md index 305b15f..23c35da 100644 --- a/docs/src/contributing.md +++ b/docs/src/contributing.md @@ -1,11 +1,8 @@ ---- -hide: - - navigation - - toc ---- Issues and pull requests are more than welcome: https://github.com/developmentseed/eoAPI/issues +You can also start **Discussions** in https://github.com/developmentseed/eoAPI/discussions + **dev install** ```bash @@ -23,7 +20,9 @@ python -m pip install "psycopg[binary,pool]" python -m pip install -e runtime/eoapi/raster["test"] # or -e runtime/eoapi/stac["test"] | -e runtime/eoapi/vector["test"] ``` -Note: services might have incompatible dependencies which you can resolve by using virtual environnement per service +!!! danger + + Python applications might have incompatible dependencies which you can resolve by using virtual environment *per application* **pre-commit** @@ -32,3 +31,13 @@ This repo is set to use `pre-commit` to run *isort*, *ruff*, *pydocstring*, *bla ```bash $ pre-commit install ``` + +### Open Source + +You can also contribute indirectly to eoAPI by helping on the sub-modules: + +- **PgSTAC** database https://github.com/stac-utils/pgstac +- **stac-fastapi**: https://github.com/stac-utils/stac-fastapi +- **titiler-pgstac**: https://github.com/stac-utils/titiler-pgstac +- **TiTiler**: https://github.com/developmentseed/titiler +- **TiPg**: https://github.com/developmentseed/tipg diff --git a/docs/src/customization.md b/docs/src/customization.md new file mode 100644 index 0000000..25ea54a --- /dev/null +++ b/docs/src/customization.md @@ -0,0 +1,67 @@ + +The eoAPI repository (https://github.com/developmentseed/eoAPI) hosts customized version of each base services. This is to show how each services can be customized and how they can work in parallel or with each other. + +--- +## eoapi.stac + +Built on [stac-fastapi.pgstac](https://github.com/stac-utils/stac-fastapi) application, adding a **`TiTilerExtension`** and a simple **`Search Viewer`**. + +The service includes: + +- Full **stac-fastapi** implementation - see [docs](http://localhost:8081/docs) if using the `docker-compose` configuration. +- Simple STAC Search **viewer** - see [viewer](http://localhost:8081/index.html) if using the `docker-compose` configuration. +- **Proxy** to the tiler endpoint for STAC Items. + +When the `TITILER_ENDPOINT` environment variable is set (pointing to the `raster` application), additional endpoints will be added to the stac-fastapi application (see: [stac/extension.py](https://github.com/developmentseed/eoAPI/blob/main/src/eoapi/stac/eoapi/stac/extension.py)): + +- `/collections/{collectionId}/items/{itemId}/tilejson.json`: Return the `raster` tilejson for an item +- `/collections/{collectionId}/items/{itemId}/viewer`: Redirect to the `raster` viewer + +

+eoapi.stac OpenAPI documentation +Metadata STAC search viewer +

+ +Code: [/runtime/eoapi/stac](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/stac) + +--- +## eoapi.raster + +The dynamic tiler deployed within `eoAPI` is built on top of [titiler-pgstac](https://github.com/stac-utils/titiler-pgstac) and [pgstac](https://github.com/stac-utils/pgstac). It enables large-scale mosaic based on the results of STAC search queries. + +The service includes all the default endpoints from **titiler-pgstac** application and: + +- `/`: a custom landing page with links to the different endpoints + +- `/mosaic/builder`: a virtual mosaic builder UI, which help creating and registering STAC Search queries + +- `/collections`: a *secret* (not in OpenAPI documentation) endpoint used in the mosaic-builder page + +- `/collections/{collection_id}/items/{item_id}/viewer`: a simple STAC Item viewer + +

+eoapi.stac OpenAPI documentation +Raster mosaic builder +STAC Item viewer +

+ + +Code: [/runtime/eoapi/raster](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/raster) + +--- +## eoapi.vector + +OGC Features and Tiles API built on top of [tipg](https://github.com/developmentseed/tipg). + +By default, the API will look for tables in the `public` schema of the database. We've also added three functions that connect to the pgSTAC schema: + +- **pg_temp.pgstac_collections_view**: Simple function which returns PgSTAC Collections +- **pg_temp.pgstac_hash**: Return features for a specific `searchId` (hash) +- **pg_temp.pgstac_hash_count**: Return the number of items per geometry for a specific `searchId` (hash) + +

+eoapi.vector OpenAPI documentation +eoapi.vector landing page +

+ +Code: [/runtime/eoapi/vector](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/vector) diff --git a/docs/src/deployment.md b/docs/src/deployment.md index 51c6f9f..5e64fed 100644 --- a/docs/src/deployment.md +++ b/docs/src/deployment.md @@ -102,11 +102,11 @@ If you don't have a k8s cluster set up on AWS or GCP then follow an IaC guide be > ⓘ The helm chart in this repo assumes your cluster has a few third-party add-ons and controllers installed. So > it's in your best interest to read through the IaC guides to understand what those defaults are -* [AWS EKS Cluster Setup](./docs/aws-eks.md) +* :octicons-link-external-16: [AWS EKS Cluster Setup](https://github.com/developmentseed/eoapi-k8s/blob/main/docs/aws-eks.md) -* [TBD: GCP GKE Cluster Setup](./docs/gcp-gke.md) - -**Helm Installation** +* :octicons-link-external-16: [TBD: GCP GKE Cluster Setup](https://github.com/developmentseed/eoapi-k8s/blob/main/docs/gcp-gke.md) + +**Helm Installation** Once you have a k8s cluster set up you can `helm install` eoAPI as follows @@ -121,7 +121,7 @@ Once you have a k8s cluster set up you can `helm install` eoAPI as follows $ export POSTGRES_USER=s00pers3cr3t $ export POSTGRES_PASSWORD=superuserfoobar $ export PGPASSWORD=foobar - + $ cd ./helm-chart $ helm install \ @@ -141,15 +141,15 @@ Once you have a k8s cluster set up you can `helm install` eoAPI as follows ```python # add the eoapi helm repo locally $ helm repo add eoapi https://devseed.com/eoapi-k8s/ - + # list out the eoapi chart versions - $ helm search repo eoapi - NAME CHART VERSION APP VERSION DESCRIPTION + $ helm search repo eoapi + NAME CHART VERSION APP VERSION DESCRIPTION eoapi/eoapi 0.1.1 0.1.0 Create a full Earth Observation API with Metada... eoapi/eoapi 0.1.2 0.1.0 Create a full Earth Observation API with Metada... - + # add the required secret overrides to an arbitrarily named `.yaml` file (`config.yaml` below) - $ cat config.yaml + $ cat config.yaml db: settings: secrets: @@ -157,7 +157,7 @@ Once you have a k8s cluster set up you can `helm install` eoAPI as follows POSTGRES_USER: "username" PGPASSWORD: "password" POSTGRES_PASSWORD: "password" - - # then run `helm install` with those overrides + + # then run `helm install` with those overrides helm install eoapi eoapi/eoapi --version 0.1.1 -f config.yaml ``` diff --git a/docs/src/intro.md b/docs/src/intro.md index 9d4da4e..19de757 100644 --- a/docs/src/intro.md +++ b/docs/src/intro.md @@ -1,8 +1,3 @@ ---- -hide: - - navigation ---- -

@@ -67,11 +62,25 @@ See [service details](./services.md) for more information. ## Getting started -- Clone the repository: `git clone https://github.com/developmentseed/eoAPI.git` -- Navigate to the project: `cd eoAPI` -- Run services with `docker compose up` -- Follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)) to load some data into eoAPI -- Checkout the [Search Viewer](http://localhost:8081/index.html), and the API documentation ([STAC Metadata](http://localhost:8081/docs), [Raster Tiles](http://localhost:8082/docs), [Vector Tiles](http://localhost:8083/api.html)) +The easiest way to start exploring the different eoAPI services is with *Docker*. Clone this repository and start the multi-container *Docker* applications using `Compose`: + +``` +git clone https://github.com/developmentseed/eoAPI.git +cd eoAPI +docker compose up +``` + +Once the application are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). + + +Then you can start exploring your dataset with: + + - the STAC Metadata service [http://localhost:8081](http://localhost:8081) + - the Raster service [http://localhost:8082](http://localhost:8082) + +!!! info + + If you've added vector dataset to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). Alternatively, you may launch the application locally: ```bash @@ -87,5 +96,7 @@ export DATABASE_URL=postgresql://username:password@0.0.0.0:5439/postgis # Conne .venv/bin/uvicorn eoapi.{SERVICE}.app:app --port 8000 --reload ``` -Note: services might have incompatible dependencies which you can resolve by using virtual environement per service +!!! danger + + Python applications might have incompatible dependencies which you can resolve by using virtual environment *per application* diff --git a/docs/src/services.md b/docs/src/services.md index c9a794f..aaf68fc 100644 --- a/docs/src/services.md +++ b/docs/src/services.md @@ -1,10 +1,3 @@ ---- - -hide: - -- navigation - ---- eoAPI combines several state-of-the-art projects to create a full Earth Observation API. Each service can be used and deployed independently but eoAPI creates the interconnections between each service: @@ -13,55 +6,109 @@ eoAPI combines several state-of-the-art projects to create a full Earth Observat - STAC Items And Mosaic Raster Tiles API built on top of https://github.com/stac-utils/titiler-pgstac - OGC Features and Vector Tiles API built on top of https://github.com/developmentseed/tipg -## Metadata +## Database -Built on [stac-fastapi.pgstac](https://github.com/stac-utils/stac-fastapi) application, adding a **`TiTilerExtension`** and a simple **`Search Viewer`**. +The STAC database is at the heart of eoAPI and is the only **mandatory** service. We use [`PgSTAC`](https://github.com/stac-utils/pgstac) Postgres schema and functions which provides functionality for STAC Filters, CQL2 search, and utilities to help manage the indexing and partitioning of STAC Collections and Items. -The service includes: +> PgSTAC is used in production to scale to hundreds of millions of STAC items. PgSTAC implements core data models and functions to provide a STAC API from a PostgreSQL database. PgSTAC is entirely within the database and does not provide an HTTP-facing API. The Stac FastAPI PgSTAC backend and Franklin can be used to expose a PgSTAC catalog. Integrating PgSTAC with any other language with PostgreSQL drivers is also possible. +> +> PgSTAC Documentation: https://stac-utils.github.io/pgstac/pgstac +> +> pyPgSTAC Documentation: https://stac-utils.github.io/pgstac/pypgstac -- Full **stac-fastapi** implementation - see [docs](http://localhost:8081/docs) if using the `docker-compose` configuration. -- Simple STAC Search **viewer** - see [viewer](http://localhost:8081/index.html) if using the `docker-compose` configuration. -- **Proxy** to the tiler endpoint for STAC Items. - -When the `TITILER_ENDPOINT` environment variable is set (pointing to the `raster` application), additional endpoints will be added to the stac-fastapi application (see: [stac/extension.py](https://github.com/developmentseed/eoAPI/blob/main/src/eoapi/stac/eoapi/stac/extension.py)): +## Metadata + +The Metadata service deployed in eoAPI is built on [stac-fastapi.pgstac](https://github.com/stac-utils/stac-fastapi) application. + +By default, the STAC metadata service will have a set of endpoint to *search* and list STAC collections and items. -- `/collections/{collectionId}/items/{itemId}/tilejson.json`: Return the `raster` tilejson for an item -- `/collections/{collectionId}/items/{itemId}/viewer`: Redirect to the `raster` viewer

- -

STAC Search viewer

+stac-fastapi

-Code: [/runtime/eoapi/stac](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/stac) +!!! example + + - https://stac.eoapi.dev landing page + + - https://stac.eoapi.dev/collections list available **Collection** + + - https://stac.eoapi.dev/collections/MAXAR_southafrica_flooding22/items list available **Items** for the `MAXAR_southafrica_flooding22` collection + + - https://stac.eoapi.dev/collections/MAXAR_southafrica_flooding22/items/36_213131033000_1040010076566100 get `36_213131033000_1040010076566100` **Item** in the `MAXAR_southafrica_flooding22` collection + + - https://stac.eoapi.dev/search list of **Items** in the catalog + + - https://stac.eoapi.dev/search?collections=MAXAR_Kahramanmaras_turkey_earthquake_23&limit=5&datetime=2023-02-06T00:00:00Z/2023-02-10T00:00:00Z list of **Items** in the catalog for the `MAXAR_Kahramanmaras_turkey_earthquake_23` collection between February 6th and 10th --- ## Raster -The dynamic tiler deployed within `eoAPI` is built on top of [titiler-pgstac](https://github.com/stac-utils/titiler-pgstac) and [pgstac](https://github.com/stac-utils/pgstac). It enables large-scale mosaic based on the results of STAC search queries. - -The service includes: +The Raster service deployed in `eoAPI` is built on top of [titiler-pgstac](https://stac-utils.github.io/titiler-pgstac/). -- Full **titiler-pgstac** implementation. +It enables Raster visualization for single STAC **Item** and large-scale (multi collections/items) mosaic based on STAC search queries.

- -

TiTiler-PgSTAC workflow

+titiler-pgstac

-Code: [/runtime/eoapi/raster](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/raster) +!!! example + + - https://raster.eoapi.dev landing page + + **Items endpoints** + + - https://raster.eoapi.dev/collections/MAXAR_southafrica_flooding22/items/36_213131033000_1040010076566100/info get Raster metadata information about **Assets** found in `36_213131033000_1040010076566100` **Item** in the `MAXAR_southafrica_flooding22` collection + + - https://raster.eoapi.dev/collections/MAXAR_southafrica_flooding22/items/36_213131033000_1040010076566100/info?assets=visual get Raster metadata information only for the `visual` **Asset** + + - https://raster.eoapi.dev/collections/MAXAR_southafrica_flooding22/items/36_213131033000_1040010076566100/map?assets=visual&minzoom=12&maxzoom=19 show the `visual` **Asset** on a Map client + + - https://raster.eoapi.dev/collections/MAXAR_southafrica_flooding22/items/36_213131033000_1040010076566100/tilejson.json?assets=ms_analytic&minzoom=13&maxzoom=17&asset_bidx=ms_analytic|8,2,1&rescale=0,4000 get a TileJSON document for the `ms_analytic` **Asset** with band combination 8,2,1 with values rescaling from 0,4000 to 0,255 + + + **Mosaic endpoints** + + - https://raster.eoapi.dev/mosaic/list list pre-registered Virtual Mosaics + + - https://raster.eoapi.dev/mosaic/ebe10c878834f8319311ad47e2472f3b/info get information about the `ebe10c878834f8319311ad47e2472f3b` mosaic + + - https://raster.eoapi.dev/mosaic/ebe10c878834f8319311ad47e2472f3b/map?assets=visual&minzoom=12&maxzoom=19 show the `ebe10c878834f8319311ad47e2472f3b` mosaic and using the `visual` **Asset** on a Map client + + - https://raster.eoapi.dev/mosaic/ebe10c878834f8319311ad47e2472f3b/tilejson.json?assets=visual&minzoom=12&maxzoom=19 get a TileJSON document for the `ebe10c878834f8319311ad47e2472f3b` mosaic and using the `visual` **Asset** --- ## Vector -OGC Features and Tiles API built on top of [tipg](https://github.com/developmentseed/tipg). +The OGC Features and (Mapbox Vector) Tiles API service deployed in `eoAPI` is built on top of [tipg](https://github.com/developmentseed/tipg)). + +It enables vector Features/Features Collection exploration and visualization for Tables stored in the Postgres database (in the `public` schema). + +

+tipg +

+ +!!! example + + - https://vector.eoapi.dev landing page + + **OGC Features** + + - https://vector.eoapi.dev/collections list available **Tables** or **Function Layers** + + - https://vector.eoapi.dev/collections/public.landsat_wrs get information about the `landsat_wrs` Table + + - https://vector.eoapi.dev/collections/public.landsat_wrs/items list items for the `landsat_wrs` Table + + **OGC Tiles** + + - https://vector.eoapi.dev/collections/public.landsat_wrs/tiles list all TileSet available for the `landsat_wrs` Table + + - https://vector.eoapi.dev/collections/public.landsat_wrs/tiles/WebMercatorQuad get `WebMercatorQuad` TileSet information for the `landsat_wrs` Table -By default, the API will look for tables in the `public` schema of the database. We've also added three functions that connect to the pgSTAC schema: + - https://vector.eoapi.dev/collections/public.landsat_wrs/viewer show the `landsat_wrs` Table on a Map client using vector tiles -- **pg_temp.pgstac_collections_view**: Simple function which returns PgSTAC Collections -- **pg_temp.pgstac_hash**: Return features for a specific `searchId` (hash) -- **pg_temp.pgstac_hash_count**: Return the number of items per geometry for a specific `searchId` (hash) + - https://vector.eoapi.dev/tileMatrixSets/WebMercatorQuad `WebMercatorQuad` TileMatrixSet information -Code: [/runtime/eoapi/vector](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/vector) \ No newline at end of file From c863e767793575c5763084dce1a0f00a7edd76ab Mon Sep 17 00:00:00 2001 From: Zachary Deziel Date: Wed, 25 Oct 2023 11:55:16 -0700 Subject: [PATCH 2/2] Review PR and fix a few typos --- README.md | 19 +++++++++---------- docs/src/contributing.md | 7 +++---- docs/src/customization.md | 7 +++---- docs/src/deployment.md | 22 +++++++++++----------- docs/src/intro.md | 20 +++++++++----------- docs/src/services.md | 14 ++++++-------- 6 files changed, 41 insertions(+), 48 deletions(-) diff --git a/README.md b/README.md index c376b72..646b5c1 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ -

-

Create a full Earth Observation API with Metadata, Raster and Vector services.

+

Create a full Earth Observation API with Metadata, Raster, and Vector services.

@@ -23,7 +22,7 @@ ## **E**arth **O**bservation **API** -`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently but `eoAPI` creates the interconnections between each service: +`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently, but `eoAPI` creates the interconnections between each service: - **pgSTAC** database [https://github.com/stac-utils/pgstac](https://github.com/stac-utils/pgstac) @@ -37,7 +36,7 @@ ## 🌍 eoAPI: An Open-Source Community Project -`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool with us. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. +`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. --- @@ -52,14 +51,14 @@ cd eoAPI docker compose up ``` -Once the application are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). +Once the applications are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). Then you can start exploring your dataset with: - the STAC Metadata service [http://localhost:8081](http://localhost:8081) - the Raster service [http://localhost:8082](http://localhost:8082) -If you've added vector dataset to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). +If you've added a vector dataset to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). Alternatively, you may launch the application locally: ```bash @@ -68,14 +67,14 @@ virtualenv .venv source .venv/bin/activate python -m pip install "psycopg[binary,pool]" uvicorn -python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac` +python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac.` export DATABASE_URL=postgresql://username:password@0.0.0.0:5439/postgis # Connect to the database of your choice .venv/bin/uvicorn eoapi.{SERVICE}.app:app --port 8000 --reload ``` -Note: services might have incompatible dependencies which you can resolve by using a virtual environment for each service +Note: services might have incompatible dependencies, which you can resolve by using a virtual environment for each service --- @@ -93,7 +92,7 @@ Finally, [eoapi-template](https://github.com/developmentseed/eoapi-template) is ## Contribution & Development -We highly value and rely on our community! Whether you're an expert or just getting started, you can make a difference. Here's how: +We highly value and rely on our community! You can make a difference whether you're an expert or just getting started. Here's how: - **Contribute**: Check out our [CONTRIBUTING.md](https://github.com/developmentseed/eoAPI/blob/main/CONTRIBUTING.md) guide to understand how you can contribute. - **Engage in Discussions**: Share your ideas, ask questions, or provide feedback through [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions). This is where most of our project conversations take place. @@ -114,4 +113,4 @@ See [contributors](https://github.com/developmentseed/eoAPI/graphs/contributors) ## Changes -See [CHANGES.md](https://github.com/developmentseed/eoAPI/blob/main/CHANGES.md). +See [CHANGES.md](https://github.com/developmentseed/eoAPI/blob/main/CHANGES.md). \ No newline at end of file diff --git a/docs/src/contributing.md b/docs/src/contributing.md index 23c35da..1300390 100644 --- a/docs/src/contributing.md +++ b/docs/src/contributing.md @@ -1,4 +1,3 @@ - Issues and pull requests are more than welcome: https://github.com/developmentseed/eoAPI/issues You can also start **Discussions** in https://github.com/developmentseed/eoAPI/discussions @@ -15,14 +14,14 @@ python -m pip install --upgrade virtualenv virtualenv .venv source .venv/bin/activate -# Install eoapi module +# Install the eoapi module python -m pip install "psycopg[binary,pool]" python -m pip install -e runtime/eoapi/raster["test"] # or -e runtime/eoapi/stac["test"] | -e runtime/eoapi/vector["test"] ``` !!! danger - Python applications might have incompatible dependencies which you can resolve by using virtual environment *per application* + Python applications might have incompatible dependencies, which you can resolve by using a virtual environment *per application* **pre-commit** @@ -40,4 +39,4 @@ You can also contribute indirectly to eoAPI by helping on the sub-modules: - **stac-fastapi**: https://github.com/stac-utils/stac-fastapi - **titiler-pgstac**: https://github.com/stac-utils/titiler-pgstac - **TiTiler**: https://github.com/developmentseed/titiler -- **TiPg**: https://github.com/developmentseed/tipg +- **TiPg**: https://github.com/developmentseed/tipg \ No newline at end of file diff --git a/docs/src/customization.md b/docs/src/customization.md index 25ea54a..9e7262d 100644 --- a/docs/src/customization.md +++ b/docs/src/customization.md @@ -1,5 +1,4 @@ - -The eoAPI repository (https://github.com/developmentseed/eoAPI) hosts customized version of each base services. This is to show how each services can be customized and how they can work in parallel or with each other. +The eoAPI repository (https://github.com/developmentseed/eoAPI) hosts customized versions of each base service. The documentation below demonstrates how each service can be customized. The eoAPI services can work in parallel or in combination with each other. --- ## eoapi.stac @@ -33,7 +32,7 @@ The service includes all the default endpoints from **titiler-pgstac** applicati - `/`: a custom landing page with links to the different endpoints -- `/mosaic/builder`: a virtual mosaic builder UI, which help creating and registering STAC Search queries +- `/mosaic/builder`: a virtual mosaic builder UI, which helps create and register STAC Search queries - `/collections`: a *secret* (not in OpenAPI documentation) endpoint used in the mosaic-builder page @@ -64,4 +63,4 @@ By default, the API will look for tables in the `public` schema of the database. eoapi.vector landing page

-Code: [/runtime/eoapi/vector](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/vector) +Code: [/runtime/eoapi/vector](https://github.com/developmentseed/eoAPI/tree/main/runtime/eoapi/vector) \ No newline at end of file diff --git a/docs/src/deployment.md b/docs/src/deployment.md index 5e64fed..95f4d6c 100644 --- a/docs/src/deployment.md +++ b/docs/src/deployment.md @@ -13,7 +13,7 @@ hide: An example of Cloud Stack is available for AWS (RDS for the database and Lambda for the APIs). -The stack is deployed by the [AWS CDK](https://aws.amazon.com/cdk/) utility. Under the hood, CDK will create the deployment packages required for AWS Lambda, upload it to AWS, and handle the creation of the Lambda and API Gateway resources. +The stack is deployed by the [AWS CDK](https://aws.amazon.com/cdk/) utility. Under the hood, CDK will create the deployment packages required for AWS Lambda, upload them to AWS, and handle the creation of the Lambda and API Gateway resources. The example commands here will deploy a CloudFormation stack called `eoAPI-staging`. @@ -38,7 +38,7 @@ The example commands here will deploy a CloudFormation stack called `eoAPI-stagi 3. Update settings - Set environment variable or hard code in `infrastructure/aws/.env` file (e.g `CDK_EOAPI_DB_PGSTAC_VERSION=0.7.1`). + Set environment variable or complex code in the `infrastructure/aws/.env` file (e.g., `CDK_EOAPI_DB_PGSTAC_VERSION=0.7.1`). To modify the size of the burstable database instance, modify `CDK_EOAPI_DB_INSTANCE_SIZE` to one of the values of [`aws_cdk.aws_ec2.InstanceSize`](https://docs.aws.amazon.com/cdk/api/v2/python/aws_cdk.aws_ec2/InstanceSize.html#instancesize). The default size is `SMALL`. @@ -47,11 +47,11 @@ The example commands here will deploy a CloudFormation stack called `eoAPI-stagi - `CDK_EOAPI_DB_PGSTAC_VERSION` is a required env (see https://github.com/stac-utils/pgstac/tags for the latest version) - - You can choose which functions to deploy by setting `CDK_EOAPI_FUNCTIONS` env (e.g `CDK_EOAPI_FUNCTIONS='["stac","raster","vector"]'`) + - You can choose which functions to deploy by setting `CDK_EOAPI_FUNCTIONS` env (e.g., `CDK_EOAPI_FUNCTIONS='["stac","raster","vector"]'`) 4. Install CDK and connect to your AWS account. This step is only necessary once per AWS account. The environment variable `CDK_EOAPI_STAGE` determines the name of the stack -(e.g. eoAPI-staging or eoAPI-production) +(e.g., eoAPI-staging or eoAPI-production) ```bash # Deploy the CDK toolkit stack into an AWS environment. CDK_EOAPI_STAGE=staging \ @@ -81,7 +81,7 @@ The example commands here will deploy a CloudFormation stack called `eoAPI-stagi CDK_EOAPI_DB_PGSTAC_VERSION=0.7.1 \ npm --prefix infrastructure/aws run cdk -- deploy eoAPI-staging - # Deploy in specific region + # Deploy in a specific region AWS_DEFAULT_REGION=eu-central-1 \ AWS_REGION=eu-central-1 \ CDK_EOAPI_STAGE=staging \ @@ -89,7 +89,7 @@ The example commands here will deploy a CloudFormation stack called `eoAPI-stagi npm --prefix infrastructure/aws run cdk -- deploy eoapi-staging --profile {my-aws-profile} ``` -If you get an error saying that the max VPC's has been reached, this means that you have hit the limit for the amount of VPCs per unique AWS account and region combination. You can change the AWS region to a region that has less VPCs and deploy again to fix this. +If you get an error saying that the max VPCs have been reached, you have hit the limit for the number of VPCs per unique AWS account and region combination. You can change the AWS region to a region with fewer VPCs and deploy again to fix this. ## Via [eoapi-k8s](https://github.com/developmentseed/eoapi-k8s) @@ -97,10 +97,10 @@ If you get an error saying that the max VPC's has been reached, this means that **Getting started** -If you don't have a k8s cluster set up on AWS or GCP then follow an IaC guide below that is relevant to you +If you still need to set up a k8s cluster on AWS or GCP, then follow an IaC guide below that is relevant to you. > ⓘ The helm chart in this repo assumes your cluster has a few third-party add-ons and controllers installed. So -> it's in your best interest to read through the IaC guides to understand what those defaults are +> It's in your best interest to read through the IaC guides to understand what those defaults are * :octicons-link-external-16: [AWS EKS Cluster Setup](https://github.com/developmentseed/eoapi-k8s/blob/main/docs/aws-eks.md) @@ -108,7 +108,7 @@ If you don't have a k8s cluster set up on AWS or GCP then follow an IaC guide be **Helm Installation** -Once you have a k8s cluster set up you can `helm install` eoAPI as follows +Once you have a k8s cluster set up, you can `helm install` eoAPI as follows 1. `helm install` from this repo's `helm-chart/` folder: @@ -142,7 +142,7 @@ Once you have a k8s cluster set up you can `helm install` eoAPI as follows # add the eoapi helm repo locally $ helm repo add eoapi https://devseed.com/eoapi-k8s/ - # list out the eoapi chart versions + # List out the eoapi chart versions $ helm search repo eoapi NAME CHART VERSION APP VERSION DESCRIPTION eoapi/eoapi 0.1.1 0.1.0 Create a full Earth Observation API with Metada... @@ -160,4 +160,4 @@ Once you have a k8s cluster set up you can `helm install` eoAPI as follows # then run `helm install` with those overrides helm install eoapi eoapi/eoapi --version 0.1.1 -f config.yaml - ``` + ``` \ No newline at end of file diff --git a/docs/src/intro.md b/docs/src/intro.md index 19de757..4af66f5 100644 --- a/docs/src/intro.md +++ b/docs/src/intro.md @@ -1,7 +1,6 @@ -

-

Create a full Earth Observation API with Metadata, Raster and Vector services.

+

Create a full Earth Observation API with Metadata, Raster, and Vector services.

@@ -9,7 +8,7 @@ ## **E**arth **O**bservation **API** -`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently but `eoAPI` creates the interconnections between each service: +`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently, but `eoAPI` creates the interconnections between each service: - **pgSTAC** database [https://github.com/stac-utils/pgstac](https://github.com/stac-utils/pgstac) @@ -23,7 +22,7 @@ ## 🌍 eoAPI: An Open-Source Community Project -`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool with us. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. +`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. --- @@ -31,7 +30,7 @@ - **Focus on your use case:** `eoAPI` is used for large-scale data processing, building geographic information systems (GIS), creating real-time data applications, climate research and environmental monitoring, machine learning model training, and more. -- **Unified Repository:** `eoAPI` provides a single, unified repository to several state-of-the-art Earth Observation (EO) data services, including Metadata search (STAC), Raster, and Vector services. This can simplify the process of accessing and working with these services. +- **Unified Repository:** `eoAPI` provides a single, unified repository for several state-of-the-art Earth Observation (EO) data services, including Metadata search (STAC), Raster, and Vector services. This can simplify the process of accessing and working with these services. - **Interoperability:** `eoAPI` is designed to enable interoperability among its included services. This can make building complex applications that leverage different types of EO data easier. @@ -56,7 +55,7 @@ See [service details](./services.md) for more information. -*Note: The documentation links referenced require lauching the application with `docker compose` or another deployment*. +*Note: The documentation links referenced require launching the application with `docker compose` or another deployment*. --- @@ -70,7 +69,7 @@ cd eoAPI docker compose up ``` -Once the application are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). +Once the applications are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). Then you can start exploring your dataset with: @@ -80,7 +79,7 @@ Then you can start exploring your dataset with: !!! info - If you've added vector dataset to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). + If you've added vector datasets to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). Alternatively, you may launch the application locally: ```bash @@ -89,7 +88,7 @@ virtualenv .venv source .venv/bin/activate python -m pip install "psycopg[binary,pool]" uvicorn -python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac` +python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac.` export DATABASE_URL=postgresql://username:password@0.0.0.0:5439/postgis # Connect to the database of your choice @@ -98,5 +97,4 @@ export DATABASE_URL=postgresql://username:password@0.0.0.0:5439/postgis # Conne !!! danger - Python applications might have incompatible dependencies which you can resolve by using virtual environment *per application* - + Python applications might have incompatible dependencies, which you can resolve by using a virtual environment *per application* \ No newline at end of file diff --git a/docs/src/services.md b/docs/src/services.md index aaf68fc..b547fc7 100644 --- a/docs/src/services.md +++ b/docs/src/services.md @@ -1,5 +1,4 @@ - -eoAPI combines several state-of-the-art projects to create a full Earth Observation API. Each service can be used and deployed independently but eoAPI creates the interconnections between each service: +eoAPI combines several state-of-the-art projects to create an entire Earth Observation API. Each service can be used and deployed independently, but eoAPI creates the interconnections between each service: - pgSTAC database https://github.com/stac-utils/pgstac - STAC API built on top of https://github.com/stac-utils/stac-fastapi @@ -8,7 +7,7 @@ eoAPI combines several state-of-the-art projects to create a full Earth Observat ## Database -The STAC database is at the heart of eoAPI and is the only **mandatory** service. We use [`PgSTAC`](https://github.com/stac-utils/pgstac) Postgres schema and functions which provides functionality for STAC Filters, CQL2 search, and utilities to help manage the indexing and partitioning of STAC Collections and Items. +The STAC database is at the heart of eoAPI and is the only **mandatory** service. We use [`PgSTAC`](https://github.com/stac-utils/pgstac) Postgres schema and functions, which provides functionality for STAC Filters, CQL2 search, and utilities to help manage the indexing and partitioning of STAC Collections and Items. > PgSTAC is used in production to scale to hundreds of millions of STAC items. PgSTAC implements core data models and functions to provide a STAC API from a PostgreSQL database. PgSTAC is entirely within the database and does not provide an HTTP-facing API. The Stac FastAPI PgSTAC backend and Franklin can be used to expose a PgSTAC catalog. Integrating PgSTAC with any other language with PostgreSQL drivers is also possible. > @@ -20,7 +19,7 @@ The STAC database is at the heart of eoAPI and is the only **mandatory** service The Metadata service deployed in eoAPI is built on [stac-fastapi.pgstac](https://github.com/stac-utils/stac-fastapi) application. -By default, the STAC metadata service will have a set of endpoint to *search* and list STAC collections and items. +By default, the STAC metadata service will have a set of endpoints to *search* and list STAC collections and items.

@@ -47,7 +46,7 @@ By default, the STAC metadata service will have a set of endpoint to *search* an The Raster service deployed in `eoAPI` is built on top of [titiler-pgstac](https://stac-utils.github.io/titiler-pgstac/). -It enables Raster visualization for single STAC **Item** and large-scale (multi collections/items) mosaic based on STAC search queries. +It enables Raster visualization for a single STAC **Item** and large-scale (multi collections/items) mosaic based on STAC search queries.

titiler-pgstac @@ -108,7 +107,6 @@ It enables vector Features/Features Collection exploration and visualization for - https://vector.eoapi.dev/collections/public.landsat_wrs/tiles/WebMercatorQuad get `WebMercatorQuad` TileSet information for the `landsat_wrs` Table - - https://vector.eoapi.dev/collections/public.landsat_wrs/viewer show the `landsat_wrs` Table on a Map client using vector tiles - - - https://vector.eoapi.dev/tileMatrixSets/WebMercatorQuad `WebMercatorQuad` TileMatrixSet information + - https://vector.eoapi.dev/collections/public.landsat_wrs/viewer shows the `landsat_wrs` Table on a Map client using vector tiles + - https://vector.eoapi.dev/tileMatrixSets/WebMercatorQuad `WebMercatorQuad` TileMatrixSet information \ No newline at end of file