-
-
Create a full Earth Observation API with Metadata, Raster and Vector services.
+Create a full Earth Observation API with Metadata, Raster, and Vector services.
@@ -23,7 +22,7 @@ ## **E**arth **O**bservation **API** -`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently but `eoAPI` creates the interconnections between each service: +`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently, but `eoAPI` creates the interconnections between each service: - **pgSTAC** database [https://github.com/stac-utils/pgstac](https://github.com/stac-utils/pgstac) @@ -37,18 +36,29 @@ ## 🌍 eoAPI: An Open-Source Community Project -`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool with us. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. +`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. --- ## Getting started -- Clone the repository: `git clone https://github.com/developmentseed/eoAPI.git` -- Navigate to the project: `cd eoAPI` -- Run services with `docker compose up` -- Follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](./demo/)) to load some data into eoAPI -- Check out the [Search Viewer](http://localhost:8081/index.html), and the API documentation ([STAC Metadata](http://localhost:8081/docs), [Raster Tiles](http://localhost:8082/docs), [Vector Tiles](http://localhost:8083/api.html)) +The easiest way to start exploring the different eoAPI services is with *Docker*. Clone this repository and start the multi-container *Docker* applications using `Compose`: + +``` +git clone https://github.com/developmentseed/eoAPI.git +cd eoAPI +docker compose up +``` + +Once the applications are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). + +Then you can start exploring your dataset with: + + - the STAC Metadata service [http://localhost:8081](http://localhost:8081) + - the Raster service [http://localhost:8082](http://localhost:8082) + +If you've added a vector dataset to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). Alternatively, you may launch the application locally: ```bash @@ -57,22 +67,22 @@ virtualenv .venv source .venv/bin/activate python -m pip install "psycopg[binary,pool]" uvicorn -python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac` +python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac.` export DATABASE_URL=postgresql://username:password@0.0.0.0:5439/postgis # Connect to the database of your choice .venv/bin/uvicorn eoapi.{SERVICE}.app:app --port 8000 --reload ``` -Note: services might have incompatible dependencies which you can resolve by using a virtual environment for each service +Note: services might have incompatible dependencies, which you can resolve by using a virtual environment for each service --- ## Deployment -This repository has current runtimes that are consistently updated with new functionality. +This repository has current runtimes that are consistently updated with new functionality. -The services can be deployed locally via docker with `docker-compose up`. The official runtimes can be launched with `docker-compose -f docker-compose.yml -f docker-compose.official.yml up stac-fastapi titiler-pgstac tipg`. +The services can be deployed locally via docker with `docker-compose up`. The official runtimes can be launched with `docker compose -f docker-compose.official.yml up stac-fastapi titiler-pgstac tipg`. Two Infrastructure as Code (IaC) repositories are available: - [eoapi-cdk](https://github.com/developmentseed/eoapi-cdk): A set of AWS CDK constructs to deploy eoAPI services @@ -82,7 +92,7 @@ Finally, [eoapi-template](https://github.com/developmentseed/eoapi-template) is ## Contribution & Development -We highly value and rely on our community! Whether you're an expert or just getting started, you can make a difference. Here's how: +We highly value and rely on our community! You can make a difference whether you're an expert or just getting started. Here's how: - **Contribute**: Check out our [CONTRIBUTING.md](https://github.com/developmentseed/eoAPI/blob/main/CONTRIBUTING.md) guide to understand how you can contribute. - **Engage in Discussions**: Share your ideas, ask questions, or provide feedback through [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions). This is where most of our project conversations take place. @@ -103,4 +113,4 @@ See [contributors](https://github.com/developmentseed/eoAPI/graphs/contributors) ## Changes -See [CHANGES.md](https://github.com/developmentseed/eoAPI/blob/main/CHANGES.md). +See [CHANGES.md](https://github.com/developmentseed/eoAPI/blob/main/CHANGES.md). \ No newline at end of file diff --git a/docker-compose.official.yml b/docker-compose.official.yml index 0aa56a7..b6f54c4 100644 --- a/docker-compose.official.yml +++ b/docker-compose.official.yml @@ -2,6 +2,9 @@ version: '3' services: stac-fastapi: + # Note: + # the official ghcr.io/stac-utils/stac-fastapi-pgstac image uses python 3.8 and uvicorn + # which is why here we use a custom Dockerfile using python 3.11 and gunicorn build: context: . dockerfile: dockerfiles/Dockerfile.stac @@ -40,7 +43,7 @@ services: # At the time of writing, rasterio and psycopg wheels are not available for arm64 arch # so we force the image to be built with linux/amd64 platform: linux/amd64 - image: ghcr.io/stac-utils/titiler-pgstac:latest + image: ghcr.io/stac-utils/titiler-pgstac:0.8.0 ports: - "${MY_DOCKER_IP:-127.0.0.1}:8082:8082" environment: @@ -86,7 +89,7 @@ services: - ./dockerfiles/scripts:/tmp/scripts tipg: - image: ghcr.io/developmentseed/tipg:latest + image: ghcr.io/developmentseed/tipg:0.4.4 ports: - "${MY_DOCKER_IP:-127.0.0.1}:8083:8083" environment: @@ -115,8 +118,7 @@ services: - ./dockerfiles/scripts:/tmp/scripts database: - container_name: eoapi.db - image: ghcr.io/stac-utils/pgstac:v0.7.10 + image: ghcr.io/stac-utils/pgstac:v0.8.1 environment: - POSTGRES_USER=username - POSTGRES_PASSWORD=password diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 3f17da6..b7d5738 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -19,12 +19,14 @@ extra: link: "https://developmentseed.org/blog" nav: - - Home: "index.md" - - Intro: "intro.md" - - Services: "services.md" - - Deployment: "deployment.md" - - Contributing: "contributing.md" - + - Home: index.md + - Getting started: + - intro.md + - Services: services.md + - Contributing: contributing.md + - Advanced User Guide: + - Customization: customization.md + - Deployment: deployment.md plugins: - search @@ -36,17 +38,17 @@ theme: palette: primary: custom accent: deep orange - features: - navigation.tabs + - navigation.tabs.sticky - navigation.instant - navigation.tracking - navigation.top - - navigation.expand + - navigation.footer + - navigation.sections favicon: img/favicon.ico logo: img/logo.png - markdown_extensions: - admonition - attr_list @@ -59,7 +61,6 @@ markdown_extensions: - pymdownx.caret: insert: false - pymdownx.details - - pymdownx.emoji - pymdownx.escapeall: hardbreak: true nbsp: true @@ -73,3 +74,6 @@ markdown_extensions: - pymdownx.tilde - toc: permalink: true + - pymdownx.emoji: + emoji_index: !!python/name:material.extensions.emoji.twemoji + emoji_generator: !!python/name:material.extensions.emoji.to_svg diff --git a/docs/src/contributing.md b/docs/src/contributing.md index 305b15f..1300390 100644 --- a/docs/src/contributing.md +++ b/docs/src/contributing.md @@ -1,11 +1,7 @@ ---- -hide: - - navigation - - toc ---- - Issues and pull requests are more than welcome: https://github.com/developmentseed/eoAPI/issues +You can also start **Discussions** in https://github.com/developmentseed/eoAPI/discussions + **dev install** ```bash @@ -18,12 +14,14 @@ python -m pip install --upgrade virtualenv virtualenv .venv source .venv/bin/activate -# Install eoapi module +# Install the eoapi module python -m pip install "psycopg[binary,pool]" python -m pip install -e runtime/eoapi/raster["test"] # or -e runtime/eoapi/stac["test"] | -e runtime/eoapi/vector["test"] ``` -Note: services might have incompatible dependencies which you can resolve by using virtual environnement per service +!!! danger + + Python applications might have incompatible dependencies, which you can resolve by using a virtual environment *per application* **pre-commit** @@ -32,3 +30,13 @@ This repo is set to use `pre-commit` to run *isort*, *ruff*, *pydocstring*, *bla ```bash $ pre-commit install ``` + +### Open Source + +You can also contribute indirectly to eoAPI by helping on the sub-modules: + +- **PgSTAC** database https://github.com/stac-utils/pgstac +- **stac-fastapi**: https://github.com/stac-utils/stac-fastapi +- **titiler-pgstac**: https://github.com/stac-utils/titiler-pgstac +- **TiTiler**: https://github.com/developmentseed/titiler +- **TiPg**: https://github.com/developmentseed/tipg \ No newline at end of file diff --git a/docs/src/customization.md b/docs/src/customization.md new file mode 100644 index 0000000..9e7262d --- /dev/null +++ b/docs/src/customization.md @@ -0,0 +1,66 @@ +The eoAPI repository (https://github.com/developmentseed/eoAPI) hosts customized versions of each base service. The documentation below demonstrates how each service can be customized. The eoAPI services can work in parallel or in combination with each other. + +--- +## eoapi.stac + +Built on [stac-fastapi.pgstac](https://github.com/stac-utils/stac-fastapi) application, adding a **`TiTilerExtension`** and a simple **`Search Viewer`**. + +The service includes: + +- Full **stac-fastapi** implementation - see [docs](http://localhost:8081/docs) if using the `docker-compose` configuration. +- Simple STAC Search **viewer** - see [viewer](http://localhost:8081/index.html) if using the `docker-compose` configuration. +- **Proxy** to the tiler endpoint for STAC Items. + +When the `TITILER_ENDPOINT` environment variable is set (pointing to the `raster` application), additional endpoints will be added to the stac-fastapi application (see: [stac/extension.py](https://github.com/developmentseed/eoAPI/blob/main/src/eoapi/stac/eoapi/stac/extension.py)): + +- `/collections/{collectionId}/items/{itemId}/tilejson.json`: Return the `raster` tilejson for an item +- `/collections/{collectionId}/items/{itemId}/viewer`: Redirect to the `raster` viewer + +
+
+
+
+
+
+
+
+
+
+
-
Create a full Earth Observation API with Metadata, Raster and Vector services.
+Create a full Earth Observation API with Metadata, Raster, and Vector services.
@@ -14,7 +8,7 @@ hide: ## **E**arth **O**bservation **API** -`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently but `eoAPI` creates the interconnections between each service: +`eoAPI` combines several *state-of-the-art* projects to create a full Earth Observation API. Each service can be used and deployed independently, but `eoAPI` creates the interconnections between each service: - **pgSTAC** database [https://github.com/stac-utils/pgstac](https://github.com/stac-utils/pgstac) @@ -28,7 +22,7 @@ hide: ## 🌍 eoAPI: An Open-Source Community Project -`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool with us. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. +`eoAPI` is proudly open-source and driven by a dedicated community of contributors. We believe in the power of open collaboration and welcome anyone to contribute, discuss, and grow this tool. Join the conversations on [GitHub Discussions](https://github.com/developmentseed/eoAPI/discussions) and make a difference in the Earth Observation realm. --- @@ -36,7 +30,7 @@ hide: - **Focus on your use case:** `eoAPI` is used for large-scale data processing, building geographic information systems (GIS), creating real-time data applications, climate research and environmental monitoring, machine learning model training, and more. -- **Unified Repository:** `eoAPI` provides a single, unified repository to several state-of-the-art Earth Observation (EO) data services, including Metadata search (STAC), Raster, and Vector services. This can simplify the process of accessing and working with these services. +- **Unified Repository:** `eoAPI` provides a single, unified repository for several state-of-the-art Earth Observation (EO) data services, including Metadata search (STAC), Raster, and Vector services. This can simplify the process of accessing and working with these services. - **Interoperability:** `eoAPI` is designed to enable interoperability among its included services. This can make building complex applications that leverage different types of EO data easier. @@ -61,17 +55,31 @@ hide: See [service details](./services.md) for more information. -*Note: The documentation links referenced require lauching the application with `docker compose` or another deployment*. +*Note: The documentation links referenced require launching the application with `docker compose` or another deployment*. --- ## Getting started -- Clone the repository: `git clone https://github.com/developmentseed/eoAPI.git` -- Navigate to the project: `cd eoAPI` -- Run services with `docker compose up` -- Follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)) to load some data into eoAPI -- Checkout the [Search Viewer](http://localhost:8081/index.html), and the API documentation ([STAC Metadata](http://localhost:8081/docs), [Raster Tiles](http://localhost:8082/docs), [Vector Tiles](http://localhost:8083/api.html)) +The easiest way to start exploring the different eoAPI services is with *Docker*. Clone this repository and start the multi-container *Docker* applications using `Compose`: + +``` +git clone https://github.com/developmentseed/eoAPI.git +cd eoAPI +docker compose up +``` + +Once the applications are *up*, you'll need to add STAC **Collections** and **Items** to the PgSTAC database. If you don't have, you can use the follow the [MAXAR open data demo](https://github.com/vincentsarago/MAXAR_opendata_to_pgstac) (or get inspired by the other [demos](https://github.com/developmentseed/eoAPI/tree/main/demo)). + + +Then you can start exploring your dataset with: + + - the STAC Metadata service [http://localhost:8081](http://localhost:8081) + - the Raster service [http://localhost:8082](http://localhost:8082) + +!!! info + + If you've added vector datasets to the `public` schema in the Postgres database, they will be available through the **Vector** service at [http://localhost:8083](http://localhost:8083). Alternatively, you may launch the application locally: ```bash @@ -80,12 +88,13 @@ virtualenv .venv source .venv/bin/activate python -m pip install "psycopg[binary,pool]" uvicorn -python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac` +python -m pip install runtime/eoapi/{SERVICE} # SERVICE should be one of `raster, vector, stac.` export DATABASE_URL=postgresql://username:password@0.0.0.0:5439/postgis # Connect to the database of your choice .venv/bin/uvicorn eoapi.{SERVICE}.app:app --port 8000 --reload ``` -Note: services might have incompatible dependencies which you can resolve by using virtual environement per service +!!! danger + Python applications might have incompatible dependencies, which you can resolve by using a virtual environment *per application* \ No newline at end of file diff --git a/docs/src/services.md b/docs/src/services.md index c9a794f..b547fc7 100644 --- a/docs/src/services.md +++ b/docs/src/services.md @@ -1,67 +1,112 @@ ---- - -hide: - -- navigation - ---- - -eoAPI combines several state-of-the-art projects to create a full Earth Observation API. Each service can be used and deployed independently but eoAPI creates the interconnections between each service: +eoAPI combines several state-of-the-art projects to create an entire Earth Observation API. Each service can be used and deployed independently, but eoAPI creates the interconnections between each service: - pgSTAC database https://github.com/stac-utils/pgstac - STAC API built on top of https://github.com/stac-utils/stac-fastapi - STAC Items And Mosaic Raster Tiles API built on top of https://github.com/stac-utils/titiler-pgstac - OGC Features and Vector Tiles API built on top of https://github.com/developmentseed/tipg -## Metadata +## Database + +The STAC database is at the heart of eoAPI and is the only **mandatory** service. We use [`PgSTAC`](https://github.com/stac-utils/pgstac) Postgres schema and functions, which provides functionality for STAC Filters, CQL2 search, and utilities to help manage the indexing and partitioning of STAC Collections and Items. -Built on [stac-fastapi.pgstac](https://github.com/stac-utils/stac-fastapi) application, adding a **`TiTilerExtension`** and a simple **`Search Viewer`**. +> PgSTAC is used in production to scale to hundreds of millions of STAC items. PgSTAC implements core data models and functions to provide a STAC API from a PostgreSQL database. PgSTAC is entirely within the database and does not provide an HTTP-facing API. The Stac FastAPI PgSTAC backend and Franklin can be used to expose a PgSTAC catalog. Integrating PgSTAC with any other language with PostgreSQL drivers is also possible. +> +> PgSTAC Documentation: https://stac-utils.github.io/pgstac/pgstac +> +> pyPgSTAC Documentation: https://stac-utils.github.io/pgstac/pypgstac -The service includes: +## Metadata -- Full **stac-fastapi** implementation - see [docs](http://localhost:8081/docs) if using the `docker-compose` configuration. -- Simple STAC Search **viewer** - see [viewer](http://localhost:8081/index.html) if using the `docker-compose` configuration. -- **Proxy** to the tiler endpoint for STAC Items. - -When the `TITILER_ENDPOINT` environment variable is set (pointing to the `raster` application), additional endpoints will be added to the stac-fastapi application (see: [stac/extension.py](https://github.com/developmentseed/eoAPI/blob/main/src/eoapi/stac/eoapi/stac/extension.py)): +The Metadata service deployed in eoAPI is built on [stac-fastapi.pgstac](https://github.com/stac-utils/stac-fastapi) application. + +By default, the STAC metadata service will have a set of endpoints to *search* and list STAC collections and items. -- `/collections/{collectionId}/items/{itemId}/tilejson.json`: Return the `raster` tilejson for an item -- `/collections/{collectionId}/items/{itemId}/viewer`: Redirect to the `raster` viewer
-
-
STAC Search viewer
+
-
-
TiTiler-PgSTAC workflow
+
+
+