🏛️ [EPIC] : Get catalyst gateway fit for operation on AWS.

[stevenj]

Summary

Fix any issues relating to differences between dev environments and AWS.

JIRA Epic

TODO

Description

AWS has some differences to our development environment.
Notably, AWS Keyspaces behaves a little different to Scylla/Cassandra, as does their implementation of Postgresql.

This Epic is to track any work required to validate cat-gateway works using those resources, in addition to our development resources.
And to capture any other tasks that may be required to be preformed to get cat-gateway fully operational on AWS.

Todo

Scylla Index DB improvements to support AWS Keyspaces

See: https://docs.aws.amazon.com/keyspaces/latest/devguide/functional-differences.html

• AWS Keyspaces does require more effort when creating schemas. Enhance the schema creation so it works transparently and reliably, regardless of if it's using Scylla DB or AQS Keyspaces. #1032
  • The aim here is "It Just Works" without further configuration both locally, and on AWS Keyspaces.
• Perform performance analysis and tuning that might be required to get maximum performance from an AWS Keyspaces database. This may require adding tunable options to how the sync process works or the number of connections in the connection pool, etc.
• Validate behavior when databases disconnect and reconnect (Both on Scylla and AWS Keyspaces) and ensure that data is not lost. However, the node should also report itself as not-ready and unable to fulfil API requests in this case. The node must not terminate if the DB is not present on start, or if it fails at any other time.

Event database improvements to support AWS RDS or Aurora

AWS does not provide "True" Postgresql, both RDS or Aurora simulate Postgresql, so there can be differences.
The tasks here are to validate that our DB implementation works correctly on the chosen AWS database backend, in addition to local test databases which are true Postgresql.

• Ensure there are no conflicts or differences in the way schema migrations get performed.
• Ensure all database queries perform as expected and are consistent with Local PostgreSQL behavior.
  • The aim here is "It Just Works" without further configuration both locally, and on AWS.
• Ensure performance is as expected, and that if there is any tuning of the database interface required, there are appropriate options to control them.

General Event DB Indexing Improvements

• 🛠️ [TASK] : Documented Public Open API Definition for cat-data-service. #887 (Completed as part of M3)
  • 🛠️ [TASK] : Find out why our OpenAPI docs display poorly catalyst-ci#316
• 🛠️ [TASK] : Disable instant decoding for pallas transaction's metadata object. hermes#336 (Completed as part of M3)
  • 🛠️ [TASK] : Update Catalyst Chain follower to utilise these new features #1037
  • After testing full integration with Catalyst Chain Follower, push the catalyst-pallas updated branch to Palls upstream.
• 🛠️ [TASK] : Purge the Volatile DB of any records that exist in the immutable DB after an immutable DB roll forward (when mithril snapshots update). #944
• 🛠️ [TASK] : Cleanup API endpoints #1029