REST/Block storage Access via OpenAPI [idea]

[bluemonk3y]

Spool topic data to block storage (i.e. S3) - as Parquet format (using schema transformation). S3 can be queried via SQL, controls can be put in place that intergrate access control to the REST endpoint (i.e. API-Gateway, Lambda tasks) for serving data. Being OpenAPI codegen (and signed libs), can be used to integrate governancve featues to control things like

• masking
• access controls etc

that is specified in the originating AsyncAPI spec for the data product

https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-select-sql-reference.html
https://github.com/lensesio/stream-reactor/tree/master/kafka-connect-aws-s3
https://dev.to/aws-builders/using-aws-s3-as-a-database-17l0)
https://aws.amazon.com/blogs/apn/archiving-amazon-msk-data-to-amazon-s3-with-the-lenses-io-s3-kafka-connect-connector/