Feature/abrfc newproduct

Dockerfile

@@ -5,7 +5,7 @@ ENV PYTHONUNBUFFERED=1
 ENV PYTEST_ADDOPTS="--color=yes"
 
 # env var for test data version to use, which should always be the most up to date
-ENV TEST_DATA_TAG=2023-05-05
+ENV TEST_DATA_TAG=2023-09-19
 
 # Note: The apt and python pip below should MOSTLY match the 
 #       cumulus-api/async-geoproc/Dockerfile to ensure the 
@@ -33,9 +33,9 @@ WORKDIR /app
 
 # Get the test data before testing it
 RUN curl -L https://github.com/USACE/cumulus-geoproc-test-data/releases/download/${TEST_DATA_TAG}/cumulus-geoproc-test-data.tar.gz \
-    --output cumulus-geoproc-test-data.tar.gz && \
-    tar -xzvf cumulus-geoproc-test-data.tar.gz && \
-    rm -f cumulus-geoproc-test-data.tar.gz
+  --output cumulus-geoproc-test-data.tar.gz && \
+  tar -xzvf cumulus-geoproc-test-data.tar.gz && \
+  rm -f cumulus-geoproc-test-data.tar.gz
 
 # Install Pip Requirements
 # This first install is the cumulus-geoproc package

README.md

@@ -12,51 +12,54 @@ Python package implementation to process incoming grids to Cloud Optimized GeoTI
   - `nbm-co-01h` --> `nbm-co-01h.py`
 - Lives in `./src/cumulus_geoproc/processors`
 - Method arguments are:
-    ```
-    src : str
-        path to input file for processing
-    dst : str, optional
-        path to temporary directory
-    acquirable: str, optional
-        acquirable slug
-    ```
+
+  ```
+  src : str
+      path to input file for processing
+  dst : str, optional
+      path to temporary directory
+  acquirable: str, optional
+      acquirable slug
+  ```
 
 - Must return list of objects (dictionary)
-    ```
-    [{
-        "filetype": str,    Matching database acquirable
-        "file": str,        Converted file
-        "datetime": str,    Valid Time, ISO format with timezone
-        "version": str      Reference Time (forecasts only), ISO format with timezone
-    }]
-    ```
+  ```
+  [{
+      "filetype": str,    Matching database acquirable
+      "file": str,        Converted file
+      "datetime": str,    Valid Time, ISO format with timezone
+      "version": str      Reference Time (forecasts only), ISO format with timezone
+  }]
+  ```
 
 ## Unit and Integration Tests
 
-The `pytest` framework was selected to test processors due to its ease in writing simple tests, readability and scalability.  The `tests` directory is under the `./src` directory next to the `cumulus_geoproc` package.  Test data for each processor lives in the GitHub repository [USACE/cumulus-geoproc-test-data](https://github.com/USACE/cumulus-geoproc-test-data).  There are many ways to run these tests but two options are configured here, Docker Container and VS Code Development Environment.  The VS Code Dev Env uses the `Dockerfile` to configure the container.
+The `pytest` framework was selected to test processors due to its ease in writing simple tests, readability and scalability. The `tests` directory is under the `./src` directory next to the `cumulus_geoproc` package. Test data for each processor lives in the GitHub repository [USACE/cumulus-geoproc-test-data](https://github.com/USACE/cumulus-geoproc-test-data). There are many ways to run these tests but two options are configured here, Docker Container and VS Code Development Environment. The VS Code Dev Env uses the `Dockerfile` to configure the container.
 
 ### Testing on the Comman Line
 
-A shell script, `docker_run.sh`, is provided to build and test the processors.  Options `-t` and `-k` run the tests and create a volume returning a report of the tests.  If the docker image has not been created, or it needs a rebuild, the `-b` option must be provided.
+A shell script, `docker_run.sh`, is provided to build and test the processors. Options `-t` and `-k` run the tests and create a volume returning a report of the tests. If the docker image has not been created, or it needs a rebuild, the `-b` option must be provided.
 
 Build and test without report generation:
+
 ```
 > ./docker_run.sh -bt
 ```
 
 Build and test with report generation:
+
 ```
 > ./docker_run.sh -btk
 ```
 
 ## VS Code Developement Container
 
-A VS Code development container configuration is provided to assist in processor and test development.  See [Create a Dev Container](https://code.visualstudio.com/docs/devcontainers/create-dev-container) to use a Docker container as a full-featured development environment.  A `devcontainer.json` configuration is provided already for development.
+A VS Code development container configuration is provided to assist in processor and test development. See [Create a Dev Container](https://code.visualstudio.com/docs/devcontainers/create-dev-container) to use a Docker container as a full-featured development environment. A `devcontainer.json` configuration is provided already for development.
 
 # Processor Test Data
 
-Test data lives in the GitHub repository [USACE/cumulus-geoproc-test-data](https://github.com/USACE/cumulus-geoproc-test-data) as a release archive `tar.gz`.  Each acquirable has its own directory with an example file(s) and `json` configuration file describing the test file(s).  `pytest` `fixtures` is used to read each configuration `json` file, aggregate them into a single fixture, and uses them to define testing.  There is a one-to-one releation between the processor and testing data.
+Test data lives in the GitHub repository [USACE/cumulus-geoproc-test-data](https://github.com/USACE/cumulus-geoproc-test-data) as a release archive `tar.gz`. Each acquirable has its own directory with an example file(s) and `json` configuration file describing the test file(s). `pytest` `fixtures` is used to read each configuration `json` file, aggregate them into a single fixture, and uses them to define testing. There is a one-to-one releation between the processor and testing data.
 
-The `tar.gz` also containes some helper scripts, `gen_markdown` and `tar_test_data.sh`.  `gen_markdown` is a Python script creating `Markdown` from each `json` configuration creating [TESTDATA.md](./TESTDATA.md).  The shell script `tar_test_data.sh` creates `cumulus-geoproc-test-data.tar.gz` if changes are added for a new release.
+The `tar.gz` also containes some helper scripts, `gen_markdown` and `tar_test_data.sh`. `gen_markdown` is a Python script creating `Markdown` from each `json` configuration creating [TESTDATA.md](./TESTDATA.md). The shell script `tar_test_data.sh` creates `cumulus-geoproc-test-data.tar.gz` if changes are added for a new release.
 
-***New release data requires the Docker file update `ENV TEST_DATA_TAG=YYYY-MM-DD`**
+**\*New release data requires the Docker file update `ENV TEST_DATA_TAG=YYYY-MM-DD`**

entrypoint.sh

@@ -5,7 +5,7 @@
 case "$1" in
     itest)
         echo "Pytest"
-        python3 -m pytest -v --html=./docs/index.html --self-contained-html
+        python3 -m pytest -v --html=./docs/index.html --self-contained-html  #-s --verbose 
         ;;
     *)
         echo "No option provided that can be used"

src/cumulus_geoproc/configurations.py

@@ -84,4 +84,4 @@
     "PRODUCT_FILE_VERSION", default="1111-11-11T11:11:11.11Z"
 )
 
-LOGGER_LEVEL: str = os.getenv("LOGGER_LEVEL", default="INFO")
+LOGGER_LEVEL: str = os.getenv("LOGGER_LEVEL", default="DEBUG")

src/cumulus_geoproc/processors/abrfc-qpf-06h.py

@@ -0,0 +1,84 @@
+"""
+# Arkansas-Red Basin River Forecast Center
+
+## QPF 06 hour total precipitation
+"""
+
+
+import os
+import sys
+import pyplugs
+
+from cumulus_geoproc import logger
+from cumulus_geoproc.utils import cgdal
+
+SUBSET_NAME = "QPF_SFC"
+SUBSET_DATATYPE = "32-bit floating-point"
+
+
+@pyplugs.register
+def process(*, src: str, dst: str = None, acquirable: str = None):
+    """
+    # Grid processor
+
+    __Requires keyword only arguments (*)__
+
+    Parameters
+    ----------
+    src : str
+        path to input file for processing
+    dst : str, optional
+        path to temporary directory
+    acquirable: str, optional
+        acquirable slug
+
+    Returns
+    -------
+    List[dict]
+    ```
+    {
+        "filetype": str,         Matching database acquirable
+        "file": str,             Converted file
+        "datetime": str,         Valid Time, ISO format with timezone
+        "version": str           Reference Time (forecast), ISO format with timezone
+    }
+    ```
+    """
+
+    try:
+        # determine the path and open the file in gdal
+        ds, src_path, dst_path = cgdal.openfileGDAL(src, dst)
+
+        ds = cgdal.findsubset(ds, [SUBSET_NAME, SUBSET_DATATYPE])
+
+        version_datetime = cgdal.getVersionDate(
+            ds, src_path, "NC_GLOBAL#creationTime", "%Y%m%d%H", "\\d{10}", False
+        )
+
+        ds, lonLL, latLL, lonUR, latUR = cgdal.geoTransform_ds(ds, SUBSET_NAME)
+        outfile_list = cgdal.subsetOutFile(
+            ds,
+            SUBSET_NAME,
+            dst_path,
+            acquirable,
+            version_datetime,
+            outputBounds=[lonLL, latUR, lonUR, latLL],
+            outputSRS="EPSG:4326",
+        )
+
+    except (RuntimeError, KeyError, Exception) as ex:
+        exc_type, exc_value, exc_traceback = sys.exc_info()
+        traceback_details = {
+            "filename": os.path.basename(exc_traceback.tb_frame.f_code.co_filename),
+            "line number": exc_traceback.tb_lineno,
+            "method": exc_traceback.tb_frame.f_code.co_name,
+            "type": exc_type.__name__,
+            "message": exc_value,
+        }
+        for k, v in traceback_details.items():
+            logger.error(f"{k}: {v}")
+
+    finally:
+        ds = None
+
+    return outfile_list