feat!: titiler upgrade, optional custom base path for apis, optional …

…cloudfront (#244) # What - adds optional configuration to deploy all endpoints with a custom root path (i.e. `/api/stac` and `/api/raster`) #229, #241 - adds support for using a cloudfront as a reverse proxy #229, #241, #245 - titiler-pgstac upgraded from 0.2.3 to 0.8.0 #239 - titiler upgrade for custom colormap configuration #243 # How tested A temporary test stack was deployed to confirm that raster-api works as expected and that the two-subdomain staging-stac and staging-raster pattern is still supported. The [pre-deploy diff action for this pr](https://github.com/NASA-IMPACT/veda-backend/actions/runs/6791564314/job/18463354484#step:11:919) confirms that the raster API lambda will be upgraded and no domain name changes will be caused.
NASA-IMPACT · Nov 13, 2023 · b5e2701 · b5e2701
2 parents 9426728 + 7072ca2
commit b5e2701
Show file tree

Hide file tree

Showing 35 changed files with 686 additions and 1,232 deletions.
diff --git a/.example.env b/.example.env
@@ -20,3 +20,12 @@ VEDA_RASTER_DATA_ACCESS_ROLE_ARN=[OPTIONAL ARN OF IAM ROLE TO BE ASSUMED BY RAST
 VEDA_RASTER_EXPORT_ASSUME_ROLE_CREDS_AS_ENVS=False
 
 VEDA_DB_PUBLICLY_ACCESSIBLE=TRUE
+
+VEDA_RASTER_ROOT_PATH=
+VEDA_STAC_ROOT_PATH=
+
+STAC_BROWSER_BUCKET=
+STAC_URL=
+CERT_ARN=
+VEDA_CLOUDFRONT=
+VEDA_CUSTOM_HOST=
diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml
@@ -116,10 +116,17 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install -e .[dev,deploy,test]
 
-      - name: Get dev environment configuration for develop branch
-        run: ./scripts/get-env.sh "veda-backend-uah-dev-env"
-
+      - name: Get environment configuration for target branch
+        run: |
+          if [ "${{ github.base_ref }}" == "main" ]; then
+            ./scripts/get-env.sh "veda-backend-uah-staging-env"
+          elif [ "${{ github.base_ref }}" == "develop" ]; then
+            ./scripts/get-env.sh "veda-backend-uah-dev-env"
+          else
+            echo "No environment associated with ${GITHUB_REF##*/} branch. Test changes against dev stack"
+            ./scripts/get-env.sh "veda-backend-uah-dev-env"
+          fi
       - name: Pre deployment CDK diff
         run: |
           echo $STAGE
-          cdk diff veda-backend-uah-dev
+          cdk diff
diff --git a/.github/workflows/tests/test_raster.py b/.github/workflows/tests/test_raster.py
@@ -33,15 +33,15 @@ def test_mosaic_api():
     assert list(resp.json()[0]) == ["id", "bbox", "assets", "collection"]
     assert resp.json()[0]["id"] == "20200307aC0853900w361030"
 
-    resp = httpx.get(f"{raster_endpoint}/mosaic/{searchid}/15/8589/12849/assets")
+    resp = httpx.get(f"{raster_endpoint}/mosaic/{searchid}/tiles/15/8589/12849/assets")
     assert resp.status_code == 200
     assert len(resp.json()) == 1
     assert list(resp.json()[0]) == ["id", "bbox", "assets", "collection"]
     assert resp.json()[0]["id"] == "20200307aC0853900w361030"
 
     z, x, y = 15, 8589, 12849
     resp = httpx.get(
-        f"{raster_endpoint}/mosaic/tiles/{searchid}/{z}/{x}/{y}",
+        f"{raster_endpoint}/mosaic/{searchid}/tiles/{z}/{x}/{y}",
         params={"assets": "cog"},
         headers={"Accept-Encoding": "br, gzip"},
         timeout=10.0,

diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -12,7 +12,7 @@ repos:
         language_version: python
 
   - repo: https://github.com/PyCQA/flake8
-    rev: 3.8.3
+    rev: 6.1.0
     hooks:
       - id: flake8
         language_version: python

diff --git a/.readme/veda-backend.drawio.svg b/.readme/veda-backend.drawio.svg
diff --git a/.readme/veda-backend.drawio.xml b/.readme/veda-backend.drawio.xml
diff --git a/README.md b/README.md
@@ -53,7 +53,8 @@ The constructs and applications in this project are configured using pydantic. T
 | Domain | `VEDA_DOMAIN` | [domain/infrastructure/config.py](domain/infrastructure/config.py) |
 | Network | `N/A` | [network/infrastructure/config.py](network/infrastructure/config.py) |
 | Raster API (TiTiler) | `VEDA_RASTER` | [raster_api/infrastructure/config.py](raster_-_api/infrastructure/config.py) |
-| STAC API | `VEDA_STAC` | [stac_api/infrastructure/config.py](stac_api/infrastructure/config.py) |
+| STAC API | `VEDA` | [stac_api/infrastructure/config.py](stac_api/infrastructure/config.py) |
+| Routes | `VEDA` | [routes/infrastructure/config.py](routes/infrastructure/config.py) |
 
 ### Deploying to the cloud
 
@@ -124,7 +125,7 @@ docker compose down
 
 > **Warning** PgSTAC records should be loaded in the database using [pypgstac](https://github.com/stac-utils/pgstac#pypgstac) for proper indexing and partitioning.
 
-The VEDA ecosystem includes tools specifially created for loading PgSTAC records and optimizing data assets. The [veda-data-pipelines](https://github.com/NASA-IMPACT/veda-data-pipelines) project provides examples of cloud pipelines that transform data to cloud optimized formats, generate STAC metadata, and submit records for publication to the veda-backend database using the [veda-stac-ingestor](https://github.com/NASA-IMPACT/veda-stac-ingestor).
+The VEDA ecosystem includes tools specifially created for loading PgSTAC records and optimizing data assets. The [veda-data-airflow](https://github.com/NASA-IMPACT/veda-data-airflow) project provides examples of cloud pipelines that transform data to cloud optimized formats, generate STAC metadata, and submit records for publication to the veda-backend database using the [veda-stac-ingestor](https://github.com/NASA-IMPACT/veda-stac-ingestor).
 
 ## Support scripts
 Support scripts are provided for manual system operations.
@@ -139,7 +140,8 @@ Support scripts are provided for manual system operations.
 | [**veda-config**](https://github.com/NASA-IMPACT/veda-config) | Configuration for viewing VEDA assets in dashboard UI  |
 | [**veda-ui**](https://github.com/NASA-IMPACT/veda-ui) | Dashboard UI for viewing and analysing VEDA assets |
 | [**veda-stac-ingestor**](https://github.com/NASA-IMPACT/veda-stac-ingestor) |  Entry-point for users/services to add new records to database |
-| [**veda-data-pipelines**](https://github.com/NASA-IMPACT/veda-data-pipelines) | Cloud optimize data assets and submit records for publication to veda-stac-ingestor |
+| [**veda-data**](https://github.com/NASA-IMPACT/veda-data) | Collection and asset discovery configuration |
+| [**veda-data-airflow**](https://github.com/NASA-IMPACT/veda-data-airflow) | Cloud optimize data assets and submit records for publication to veda-stac-ingestor |
 | [**veda-docs**](https://github.com/NASA-IMPACT/veda-docs) | Documentation repository for end users of VEDA ecosystem data and tools |
 
 ## VEDA usage examples

diff --git a/app.py b/app.py
@@ -10,6 +10,7 @@
 from network.infrastructure.construct import VpcConstruct
 from permissions_boundary.infrastructure.construct import PermissionsBoundaryAspect
 from raster_api.infrastructure.construct import RasterApiLambdaConstruct
+from routes.infrastructure.construct import CloudfrontDistributionConstruct
 from stac_api.infrastructure.construct import StacApiLambdaConstruct
 
 app = App()
@@ -57,18 +58,29 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
 raster_api = RasterApiLambdaConstruct(
     veda_stack,
     "raster-api",
+    stage=veda_app_settings.stage_name(),
     vpc=vpc.vpc,
     database=database,
-    domain_name=domain.raster_domain_name,
+    domain=domain,
 )
 
 stac_api = StacApiLambdaConstruct(
     veda_stack,
     "stac-api",
+    stage=veda_app_settings.stage_name(),
     vpc=vpc.vpc,
     database=database,
     raster_api=raster_api,
-    domain_name=domain.stac_domain_name,
+    domain=domain,
+)
+
+veda_routes = CloudfrontDistributionConstruct(
+    veda_stack,
+    "routes",
+    stage=veda_app_settings.stage_name(),
+    raster_api_id=raster_api.raster_api.api_id,
+    stac_api_id=stac_api.stac_api.api_id,
+    region=veda_app_settings.cdk_default_region,
 )
 
 # TODO this conditional supports deploying a second set of APIs to a separate custom domain and should be removed if no longer necessary
@@ -83,6 +95,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
     alt_raster_api = RasterApiLambdaConstruct(
         veda_stack,
         "alt-raster-api",
+        stage=veda_app_settings.stage_name(),
         vpc=vpc.vpc,
         database=database,
         domain_name=alt_domain.raster_domain_name,
@@ -91,6 +104,7 @@ def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
     alt_stac_api = StacApiLambdaConstruct(
         veda_stack,
         "alt-stac-api",
+        stage=veda_app_settings.stage_name(),
         vpc=vpc.vpc,
         database=database,
         raster_api=raster_api,

diff --git a/domain/infrastructure/config.py b/domain/infrastructure/config.py
@@ -14,6 +14,13 @@ class vedaDomainSettings(BaseSettings):
     hosted_zone_name: Optional[str] = Field(
         None, description="Custom domain name, i.e. veda-backend.xyz"
     )
+    create_custom_subdomains: bool = Field(
+        False,
+        description=(
+            "When true and hosted zone config is provided, create a unique subdomain for stac and raster apis. "
+            "For example <stage>-stac.<hosted_zone_name> and <stage>-raster.<hosted_zone_name>"
+        ),
+    )
     api_prefix: Optional[str] = Field(
         None,
         description=(

diff --git a/domain/infrastructure/construct.py b/domain/infrastructure/construct.py
@@ -30,10 +30,8 @@ def __init__(
         self.stac_domain_name = None
         self.raster_domain_name = None
 
-        if (
-            veda_domain_settings.hosted_zone_id
-            and veda_domain_settings.hosted_zone_name
-        ):
+        if veda_domain_settings.create_custom_subdomains:
+
             # If alternative custom domain provided, use it instead of the default
             if alt_domain is True:
                 hosted_zone_name = veda_domain_settings.alt_hosted_zone_name
@@ -115,5 +113,7 @@ def __init__(
                 value=f"https://{raster_url_prefix}.{hosted_zone_name}/docs",
             )
             CfnOutput(
-                self, "stac-api", value=f"https://{stac_url_prefix}.{hosted_zone_name}/"
+                self,
+                "stac-api",
+                value=f"https://{stac_url_prefix}.{hosted_zone_name}/",
             )
diff --git a/raster_api/infrastructure/config.py b/raster_api/infrastructure/config.py
@@ -45,40 +45,45 @@ class vedaRasterSettings(BaseSettings):
     timeout: int = 30  # seconds
     memory: int = 8000  # Mb
 
-    enable_mosaic_search: bool = Field(
+    raster_enable_mosaic_search: bool = Field(
         False,
         description="Deploy the raster API with the mosaic/list endpoint TRUE/FALSE",
     )
-    pgstac_secret_arn: Optional[str] = Field(
+    raster_pgstac_secret_arn: Optional[str] = Field(
         None,
         description="Name or ARN of the AWS Secret containing database connection parameters",
     )
 
-    data_access_role_arn: Optional[str] = Field(
+    raster_data_access_role_arn: Optional[str] = Field(
         None,
         description="Resource name of role permitting access to specified external S3 buckets",
     )
 
-    export_assume_role_creds_as_envs: Optional[bool] = Field(
+    raster_export_assume_role_creds_as_envs: Optional[bool] = Field(
         False,
         description="enables 'get_gdal_config' flow to export AWS credentials as os env vars",
     )
 
-    aws_request_payer: Optional[str] = Field(
+    raster_aws_request_payer: Optional[str] = Field(
         None,
         description="Set optional global parameter to 'requester' if the requester agrees to pay S3 transfer costs",
     )
 
-    path_prefix: Optional[str] = Field(
+    raster_root_path: str = Field(
         "",
-        description="Optional path prefix to add to all api endpoints",
+        description="Optional root path for all api endpoints",
+    )
+
+    custom_host: str = Field(
+        None,
+        description="Complete url of custom host including subdomain. When provided, override host in api integration",
     )
 
     class Config:
         """model config"""
 
         env_file = ".env"
-        env_prefix = "VEDA_RASTER_"
+        env_prefix = "VEDA_"
 
 
 veda_raster_settings = vedaRasterSettings()
diff --git a/raster_api/infrastructure/construct.py b/raster_api/infrastructure/construct.py
@@ -1,5 +1,7 @@
 """CDK Constrcut for a Lambda based TiTiler API with pgstac extension."""
 import os
+import typing
+from typing import Optional
 
 from aws_cdk import (
     CfnOutput,
@@ -16,6 +18,9 @@
 
 from .config import veda_raster_settings
 
+if typing.TYPE_CHECKING:
+    from domain.infrastructure.construct import DomainConstruct
+
 
 class RasterApiLambdaConstruct(Construct):
     """CDK Constrcut for a Lambda based TiTiler API with pgstac extension."""
@@ -24,10 +29,12 @@ def __init__(
         self,
         scope: Construct,
         construct_id: str,
+        stage: str,
         vpc,
         database,
         code_dir: str = "./",
-        domain_name: aws_apigatewayv2_alpha.DomainName = None,
+        # domain_name: aws_apigatewayv2_alpha.DomainName = None,
+        domain: Optional["DomainConstruct"] = None,
         **kwargs,
     ) -> None:
         """."""
@@ -62,33 +69,45 @@ def __init__(
 
         veda_raster_function.add_environment(
             "VEDA_RASTER_ENABLE_MOSAIC_SEARCH",
-            str(veda_raster_settings.enable_mosaic_search),
+            str(veda_raster_settings.raster_enable_mosaic_search),
         )
 
         veda_raster_function.add_environment(
             "VEDA_RASTER_PGSTAC_SECRET_ARN", database.pgstac.secret.secret_full_arn
         )
 
         veda_raster_function.add_environment(
-            "VEDA_RASTER_PATH_PREFIX", veda_raster_settings.path_prefix
+            "VEDA_RASTER_ROOT_PATH", veda_raster_settings.raster_root_path
         )
 
         # Optional AWS S3 requester pays global setting
-        if veda_raster_settings.aws_request_payer:
+        if veda_raster_settings.raster_aws_request_payer:
             veda_raster_function.add_environment(
-                "AWS_REQUEST_PAYER", veda_raster_settings.aws_request_payer
+                "AWS_REQUEST_PAYER", veda_raster_settings.raster_aws_request_payer
+            )
+
+        integration_kwargs = dict(handler=veda_raster_function)
+        if veda_raster_settings.custom_host:
+            integration_kwargs[
+                "parameter_mapping"
+            ] = aws_apigatewayv2_alpha.ParameterMapping().overwrite_header(
+                "host",
+                aws_apigatewayv2_alpha.MappingValue(veda_raster_settings.custom_host),
             )
 
         raster_api_integration = (
             aws_apigatewayv2_integrations_alpha.HttpLambdaIntegration(
-                construct_id, veda_raster_function
+                construct_id,
+                **integration_kwargs,
             )
         )
 
         domain_mapping = None
-        if domain_name:
+        # Legacy method to use a custom subdomain for this api (i.e. <stage>-raster.<domain-name>.com)
+        # If using a custom root path and/or a proxy server, do not use a custom subdomain
+        if domain and domain.raster_domain_name:
             domain_mapping = aws_apigatewayv2_alpha.DomainMappingOptions(
-                domain_name=domain_name
+                domain_name=domain.raster_domain_name
             )
 
         self.raster_api = aws_apigatewayv2_alpha.HttpApi(
@@ -112,12 +131,12 @@ def __init__(
         )
 
         # Optional use sts assume role with GetObject permissions for external S3 bucket(s)
-        if veda_raster_settings.data_access_role_arn:
+        if veda_raster_settings.raster_data_access_role_arn:
             # Get the role for external data access
             data_access_role = aws_iam.Role.from_role_arn(
                 self,
                 "data-access-role",
-                veda_raster_settings.data_access_role_arn,
+                veda_raster_settings.raster_data_access_role_arn,
             )
 
             # Allow this lambda to assume the data access role
@@ -128,12 +147,12 @@ def __init__(
 
             veda_raster_function.add_environment(
                 "VEDA_RASTER_DATA_ACCESS_ROLE_ARN",
-                veda_raster_settings.data_access_role_arn,
+                veda_raster_settings.raster_data_access_role_arn,
             )
 
         # Optional configuration to export assume role session into lambda function environment
-        if veda_raster_settings.export_assume_role_creds_as_envs:
+        if veda_raster_settings.raster_export_assume_role_creds_as_envs:
             veda_raster_function.add_environment(
                 "VEDA_RASTER_EXPORT_ASSUME_ROLE_CREDS_AS_ENVS",
-                str(veda_raster_settings.export_assume_role_creds_as_envs),
+                str(veda_raster_settings.raster_export_assume_role_creds_as_envs),
             )
diff --git a/raster_api/runtime/Dockerfile b/raster_api/runtime/Dockerfile
@@ -3,7 +3,7 @@ FROM public.ecr.aws/sam/build-python3.9:latest
 WORKDIR /tmp
 
 COPY raster_api/runtime /tmp/raster
-RUN pip install "mangum>=0.14,<0.15" /tmp/raster["psycopg-binary"] -t /asset --no-binary pydantic 
+RUN pip install "mangum>=0.14,<0.15" /tmp/raster["psycopg-binary"] -t /asset --no-binary pydantic
 RUN rm -rf /tmp/raster
 
 # # Reduce package size and remove useless files
@@ -15,4 +15,4 @@ RUN rm -rdf /asset/numpy/doc/ /asset/boto3* /asset/botocore* /asset/bin /asset/g
 
 COPY raster_api/runtime/handler.py /asset/handler.py
 
-CMD ["echo", "hello world"]
+CMD ["echo", "hello world"]
diff --git a/raster_api/runtime/handler.py b/raster_api/runtime/handler.py
@@ -6,13 +6,24 @@
 
 from mangum import Mangum
 from src.app import app
+from src.config import ApiSettings
 from src.monitoring import logger, metrics, tracer
 
+from titiler.pgstac.db import connect_to_db
+
+settings = ApiSettings()
+
 logging.getLogger("mangum.lifespan").setLevel(logging.ERROR)
 logging.getLogger("mangum.http").setLevel(logging.ERROR)
 
 
-handler = Mangum(app, lifespan="off")
+@app.on_event("startup")
+async def startup_event() -> None:
+    """Connect to database on startup."""
+    await connect_to_db(app, settings=settings.load_postgres_settings())
+
+
+handler = Mangum(app, lifespan="off", api_gateway_base_path=app.root_path)
 
 if "AWS_EXECUTION_ENV" in os.environ:
     loop = asyncio.get_event_loop()