Merge branch 'feature/es-phase-2' into cumulus-std

CHANGELOG.md

@@ -6,6 +6,19 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 
 ## Phase 2 Release
 
+### Migration Notes
+
+#### CUMULUS-3833 Migration of ReconciliationReports from DynamoDB to Postgres after Cumulus is upgraded.
+
+To invoke the Lambda and start the ReconciliationReport migration, you can use the AWS Console or CLI:
+
+```bash
+aws lambda invoke --function-name $PREFIX-ReconciliationReportMigration $OUTFILE
+```
+
+- `PREFIX` is your Cumulus deployment prefix.
+- `OUTFILE` (**optional**) is the filepath where the Lambda output will be saved.
+
 ### Replace ElasticSearch Phase 2
 
 - **CUMULUS-3229**
@@ -26,20 +39,139 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
     Elasticsearch
   - Update `@cumlus/api/ecs/async-operation` to not update Elasticsearch index when
     reporting status of async operation
+- **CUMULUS-3806**
+  - Update `@cumulus/db/search` to allow for ordered collation as a
+    dbQueryParameter
+  - Update `@cumulus/db/search` to allow `dbQueryParameters.limit` to be set to
+    `null` to allow for optional unlimited page sizes in search results
+  - Update/add type annotations/logic fixes to `@cumulus/api` reconciliation report code
+  - Annotation/typing fixes to `@cumulus/cmr-client`
+  - Typing fixes to `@cumulus/db`
+  - Re-enable Reconciliation Report integration tests
+  - Update `@cumulus/client/CMR.getToken` to throw if a non-launchpad token is requested without a username
+  - Update `Inventory` and `Granule Not Found` reports to query postgreSQL
+    database instead of elasticsearch
+  - Update `@cumulus/db/lib/granule.getGranulesByApiPropertiesQuery` to
+    allow order by collation to be optionally specified
+  - Update `@cumulus/db/lib/granule.getGranulesByApiPropertiesQuery` to
+    be parameterized and include a modifier on `temporalBoundByCreatedAt`
+  - Remove endpoint call to and all tests for Internal Reconciliation Reports
+    and updated API to throw an error if report is requested
+  - Update Orca reconciliation reports to pull granules for comparison from
+    postgres via `getGranulesByApiPropertiesQuery`
+- **CUMULUS-3837**
+  - Added `reconciliation_reports` table in RDS, including indexes
+  - Created pg model, types, and translation for `reconciliationReports` in `@cumulus/db`
+- **CUMULUS-3833**
+  - Created api types for `reconciliation_reports` in `@cumulus/types/api`
+  - Updated reconciliation reports lambda to write to new RDS table instead of Dynamo
+  - Updated `@cumulus/api/endpoints/reconciliation-reports` `getReport` and `deleteReport` to work with the new RDS table instead of Dynamo
+- **CUMULUS-3718**
+  - Updated `reconciliation_reports` list api endpoint and added `ReconciliationReportSearch` class to query postgres
+  - Added `reconciliationReports` type to stats endpoint, so `aggregate` query will work for reconciliation reports
+- **CUMULUS-3859**
+  - Updated `@cumulus/api/bin/serveUtils` to no longer add records to ElasticSearch
 
 ## [Unreleased]
 
+### Breaking Changes
+
+- **CUMULUS-3698**
+  - GranuleSearch retrieving files/execution is toggled
+      by setting "includeFullRecord" field to 'true' in relevant api endpoint params
+  - GranuleSearch does *not* retrieve files/execution by default unless includeFullRecord is set to 'true'
+  - @cumulus/db function getExecutionArnByGranuleCumulusId is removed. To replace this function use getExecutionInfoByGranuleCumulusId with parameter executionColumns set to ['arn'] or unset (['arn'] is the default argument)
+- **CUMULUS-2564**
+  - Updated `sync-granule` task to add `useGranIdPath` as a configuration flag.
+    This modifies the task behavior to stage granules to
+    `<staging_path>/<collection_id>/<md5_granuleIdHash>` to allow for better S3
+    partitioning/performance for large collections.
+    Because of this benefit
+    the default has been set to `true`, however as sync-granules relies on
+    object name collision, this configuration changes the duplicate collision
+    behavior of sync-granules to be per-granule-id instead of per-collection
+    when active.
+    If the prior behavior is desired, please add `"useGranIdPath": true` to your
+    task config in your workflow definitions that use `sync-granule`.
+
+
+### Added
+
+- **CUMULUS-3698**
+  - GranuleSearch now can retrieve associated files for granules
+  - GranuleSearch now can retrieve latest associated execution for granules
+- **CUMULUS-3919**
+  - Added terraform variables `disableSSL` and `rejectUnauthorized` to `tf-modules/cumulus-rds-tf` module.
+
+### Changed
+
+- **CUMULUS-3931**
+  - Add `force_new_deployment` to `cumulus_ecs_service` to allow users to force
+    new task deployment on terraform redeploy.   See docs for more details:
+    https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ecs_service#force_new_deployment"
+
 ### Fixed
 
-- **CUMULUS-3846**
-  - improve reliability of unit tests
-    - tests for granules api get requests separated out to new file
-    - cleanup of granule database resources to ensure no overlap
-    - ensure uniqueness of execution names from getWorkflowNameIntersectFromGranuleIds
-    - increase timeout in aws-client tests
+- **CUMULUS-3876**
+  - Fixed `s3-replicator` lambda cross region write failure
+  - Added `target_region` variable to `tf-modules/s3-replicator` module
+
+## [v19.1.0] 2024-10-07
+
+### Migration Notes
+
+This release contains changes listed here as well as changes listed in v19.0.0,
+despite v19.0.0 being deprecated. Please review Changelog entries and Migration Notes for
+each Cumulus version between your current version and v19.1.0 as normal.
+
+### Added
+
+- **CUMULUS-3020**
+  - Updated sfEventSqsToDbRecords to allow override of the default value
+   (var.rds_connection_timing_configuration.acquireTimeoutMillis / 1000) + 60)
+   via a key 'sfEventSqsToDbRecords' on `var.lambda_timeouts` on the main cumulus module/archive module
+
+  **Please note** - updating this configuration is for adavanced users only.  Value changes will modify the visibility
+  timeout on `sfEventSqsToDbRecordsDeadLetterQueue` and `sfEventSqsToDbRecordsInputQueue` and may lead to system
+  instability.
+
+- **CUMULUS-3756**
+  - Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
+  - This is to allow files matching specified regex to be excluded when updating the Related URLs list
+  - Defaults to the current behavior of excluding no files.
+- **CUMULUS-3773**
+  - Added sftpFastDownload configuration to SyncGranule task.
+  - Updated `@cumulus/sftp-client` and `@cumulus/ingest/SftpProviderClient` to support both regular and fastDownload.
+  - Added sftp support to FakeProvider
+  - Added sftp integration test
+
+### Changed
+
+- **CUMULUS-3928**
+  - updated publish scripting to use [email protected] for user email
+  - updated publish scripting to use esm over common import of latest-version
+  - updated bigint testing to remove intermitted failure source.
+  - updated postgres dependency version
+- **CUMULUS-3838**
+  - Updated python dependencies to latest:
+    - cumulus-process-py 1.4.0
+    - cumulus-message-adapter-python 2.3.0
+- **CUMULUS-3906**
+  - Bumps example ORCA deployment to version v10.0.1.
+
+### Fixed
+
+- **CUMULUS-3904**
+  - Passed sqs_message_consumer_watcher_message_limit and sqs_message_consumer_watcher_time_limit through the cumulus terraform module to the ingest terraform module.
+- **CUMULUS-3902**
+  - Update error handling to use AWS SDK V3 error classes instead of properties on js objects
 
 ## [v19.0.0] 2024-08-28
 
+### Deprecated
+This release has been deprecated in favor of the 18.5->19.1 release series. The changes
+listed here are still valid and also contained in the v19.1.0 release and beyond.
+
 ### Breaking Changes
 
 - This release includes `Replace ElasicSearch Phase 1` updates, we no longer save `collection/granule/execution` records to
@@ -97,6 +229,85 @@ ElasticSearch, the `collections/granules/executions` API endpoints are updated t
 - **CUMULUS-3792**
   - Added database indexes to improve search performance
 
+## [v18.5.1] 2024-10-25
+
+**Please note** changes in v18.5.1 may not yet be released in future versions, as this
+is a backport/patch release on the v18.5.x series of releases.  Updates that are
+included in the future will have a corresponding CHANGELOG entry in future releases.
+
+### Added
+
+- **CUMULUS-3773**
+  - Added sftpFastDownload configuration to SyncGranule task.
+  - Updated `@cumulus/sftp-client` and `@cumulus/ingest/SftpProviderClient` to support both regular and fastDownload.
+  - Added sftp support to FakeProvider
+  - Added sftp integration test
+- **CUMULUS-3756**
+  - Added excludeFileRegex configuration to UpdateGranulesCmrMetadataFileLinks
+  - This is to allow files matching specified regex to be excluded when updating the Related URLs list
+  - Defaults to the current behavior of excluding no files.
+- **CUMULUS-3919**
+  - Added terraform variables `disableSSL` and `rejectUnauthorized` to `tf-modules/cumulus-rds-tf` module.
+
+### Changed
+
+- **CUMULUS-3928**
+  - updated publish scripting to use [email protected] for user email
+  - updated publish scripting to use esm over common import of latest-version
+  - updated bigint testing to remove intermitted failure source.
+  - updated postgres dependency version
+- **CUMULUS-3838**
+  - Updated python dependencies to latest:
+    - cumulus-process-py 1.4.0
+    - cumulus-message-adapter-python 2.3.0
+
+### Fixed
+
+- **CUMULUS-3902**
+  - Update error handling to use AWS SDK V3 error classes instead of properties
+    on js objects
+
+## [v18.5.0] 2024-10-03
+
+### Migration Notes
+
+#### CUMULUS-3536 Upgrading from Aurora Serverless V1 to V2
+
+- The updates in CUMULUS-3536 require an upgrade of the postgres database.
+  Please follow [Upgrading from Aurora Serverless V1 to V2]
+  (https://nasa.github.io/cumulus/docs/next/upgrade-notes/serverless-v2-upgrade)
+
+### Added
+
+- **CUMULUS-3536**
+  - Added `rejectUnauthorized` = false to db-provision-user-database as the Lambda
+    does not have the Serverless v2 SSL certifications installed.
+
+### Changed
+
+- **CUMULUS-3725**
+  - Updated the default parameter group for `cumulus-rds-tf` to set `force_ssl`
+    to 0. This setting for the Aurora Serverless v2 database allows non-SSL
+    connections to the database, and is intended to be a temporary solution
+    until Cumulus has been updated to import the RDS rds-ca-rsa2048-g1 CA bundles in Lambda environments.
+    See [CUMULUS-3724](https://bugs.earthdata.nasa.gov/browse/CUMULUS-3724).
+
+### Fixed
+
+- **CUMULUS-3901**
+  - Fix error checking in @cumulus/errors to use Error.name in addition to Error.code
+- **CUMULUS-3824**
+  - Added the missing double quote in ecs_cluster autoscaling cf template
+- **CUMULUS-3846**
+  - improve reliability of unit tests
+    - tests for granules api get requests separated out to new file
+    - cleanup of granule database resources to ensure no overlap
+    - ensure uniqueness of execution names from getWorkflowNameIntersectFromGranuleIds
+    - increase timeout in aws-client tests
+- **Snyk**
+  - Upgraded moment from 2.29.4 to 2.30.1
+  - Upgraded pg from ~8.10 to ~8.12
+
 ## [v18.4.0] 2024-08-16
 
 ### Migration Notes
@@ -8069,8 +8280,12 @@ Note: There was an issue publishing 1.12.0. Upgrade to 1.12.1.
 
 ## [v1.0.0] - 2018-02-23
 
-[Unreleased]: https://github.com/nasa/cumulus/compare/v19.0.0...HEAD
-[v19.0.0]: https://github.com/nasa/cumulus/compare/v18.4.0...v19.0.0
+
+[Unreleased]: https://github.com/nasa/cumulus/compare/v19.1.0...HEAD
+[v19.1.0]: https://github.com/nasa/cumulus/compare/v19.0.0...v19.1.0
+[v19.0.0]: https://github.com/nasa/cumulus/compare/v18.5.0...v19.0.0
+[v18.5.1]: https://github.com/nasa/cumulus/compare/v18.5.0...v18.5.1
+[v18.5.0]: https://github.com/nasa/cumulus/compare/v18.4.0...v18.5.0
 [v18.4.0]: https://github.com/nasa/cumulus/compare/v18.3.4...v18.4.0
 [v18.3.4]: https://github.com/nasa/cumulus/compare/v18.3.3...v18.3.4
 [v18.3.3]: https://github.com/nasa/cumulus/compare/v18.3.2...v18.3.3

audit-ci.json

@@ -3,17 +3,7 @@
   "pass-enoaudit": true,
   "retry-count": 20,
   "allowlist": [
-    {
-      "GHSA-776f-qx25-q3cc": {
-        "active": true,
-        "expiry": "1 July 2023 11:00"
-      }
-    },
-    "cacheable-request",
-    "http-cache-semantics",
-    "lodash.pick",
-    "hoek",
-    "semver",
-    "axios"
+    "jsonpath-plus",
+    "semver"
   ]
 }

bamboo/bootstrap-publish.sh

@@ -17,3 +17,4 @@ apt-get install -y jq rsync zip
 echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > ~/.npmrc
 
 git config --global user.name "Bamboo CI"
+git config --global user.email "[email protected]"

bamboo/bootstrap-tf-deployment.sh

@@ -98,5 +98,3 @@ echo "Deploying Cumulus example to $DEPLOYMENT"
   -var "pdr_node_name_provider_bucket=$PDR_NODE_NAME_PROVIDER_BUCKET" \
   -var "rds_admin_access_secret_arn=$RDS_ADMIN_ACCESS_SECRET_ARN" \
   -var "orca_db_user_password=$ORCA_DATABASE_USER_PASSWORD" \
-  -var "orca_s3_access_key=$AWS_ACCESS_KEY_ID" \
-  -var "orca_s3_secret_key=$AWS_SECRET_ACCESS_KEY" \

bamboo/npm-tag.js

@@ -3,7 +3,10 @@
 'use strict';
 
 const fs = require('fs-extra');
-const getLatestVersion = require('latest-version');
+async function importGetLatestVersion() {
+  const { default: getLatestVersion } = await import('latest-version');
+  return getLatestVersion;
+}
 const semver = require('semver');
 
 // Given a git tag (process.env.GIT_TAG), this function determines what NPM
@@ -26,6 +29,7 @@ const semver = require('semver');
 //   tagged "latest" in NPM then this is a new "latest" release, and "latest" is
 //   returned.
 async function getNpmTag() {
+  const getLatestVersion = await importGetLatestVersion();
   const thisTag = process.env.GIT_TAG;
   if (!thisTag) throw new Error('Version is not set');
 

bamboo/select-stack.js

@@ -34,6 +34,7 @@ function determineIntegrationTestStackName(cb) {
     'Roger Kwarteng': 'rkwarten-ci',
     'Paul Pilone': 'ppilone-ci',
     'Tim Clark': 'teclark-ci',
+    'Bryan Wexler': 'bwexler-ci',
   };
 
   return git('.').log({ '--max-count': '1' }, (e, r) => {

bamboo/set-bamboo-env-variables.sh

@@ -39,6 +39,7 @@ declare -a param_list=(
   "bamboo_SECRET_ORCA_DATABASE_USER_PASSWORD"
   "bamboo_SECRET_PROVIDER_FTP_PORT"
   "bamboo_SECRET_PROVIDER_HTTP_PORT"
+  "bamboo_SECRET_PROVIDER_SFTP_PORT"
   "bamboo_SECRET_RDS_ADMIN_ACCESS_SECRET_ARN"
   "bamboo_SECRET_RDS_SECURITY_GROUP"
   "bamboo_SECRET_SECURITY_GROUP"

docs/deployment/README.md

@@ -263,7 +263,7 @@ $ aws dynamodb create-table \
 
 Cumulus requires a [PostgreSQL compatible database](../deployment/postgres-database-deployment.md) cluster deployed to AWS. We suggest utilizing [RDS](https://docs.aws.amazon.com/rds/index.html). For further guidance about what type of RDS database to use, please [see the guide on choosing and configuring your RDS database](./choosing_configuring_rds.md).
 
-Cumulus provides a default [template and RDS cluster module](../deployment/postgres-database-deployment.md) utilizing Aurora Serverless.
+Cumulus provides a default [template and RDS cluster module](../deployment/postgres-database-deployment.md) utilizing Aurora Serverless V2.
 
 However, Core intentionally provides a "bring your own" approach, and any well-planned cluster setup should work, given the following: