Switch from Backblaze B2 to Amazon S3

[anthonyfok]

Switch from Backblaze B2 to our own Amazon S3 bucket (private bucket for now, optionally public bucket in the future. once the latter is authorized, or switch to other alternatives in the interim if necessary. This involves at least the following repos:

• OpenDRR/boundaries
• OpenDRR/opendrr-api
• OpenDRR/opendrr-data-store → requires public S3 bucket, moved to Upload GeoPackage files (compressed or not) to public S3 bucket opendrr-data-store#47

Thanks to @drotheram for the reminder while reviewing #105 on 2021-06-09. While the use of Backblaze B2 had been very helpful in our initial exploration, it is definitely a stop-gap measure, and we need to make sure all these uses are replaced before our official release.

Tasks

Sprint 36:

• Create S3 bucket opendrr-api-prebuilt-cache-1 in "Sandbox"
  • Set Bucket policy
  • Create IAM users/groups/policies (AWS Access key ID with write permission, etc.)
• Edit OpenDRR/boundaries GitHub workflow gpkg-to-pgdump.yml to upload to S3
  • Test: https://github.com/anthonyfok/boundaries/runs/2847712508?check_suite_focus=true
  • Create pull request: gpkg-to-pgdump: Upload to Amazon S3 bucket too boundaries#21
• Write Notes on Amazon S3 in the wiki
• add_data.sh: Use aws s3 cp or aws s3 sync to fetch opendrr-boundaries.dump
• python/Docker: Install aws CLI tool
• sample.env: Provide a working set of Add placeholders for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

Sprint 37:

• Create S3 bucket in "Dev"
• Create a ticket for creating programmatic access IAM user(s)
• Adjust gpkg-to-pgdump: Upload to Amazon S3 bucket too boundaries#21, add_data: Download opendrr-boundaries.dump from S3 #121 and GitHub secrets as necessary.
• Test

[anthonyfok]

We came to the realization recently that GitHub release assets likely can replace the use of both Backblaze B2 and Amazon S3 for most of our needs. Let's revisit this issue in the future for cases where we really do need to use Amazon S3.