[help] Using S3 cloud storage with temporary AWS credentials

[solmos]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I would like to use the AWS cloud storage functionality in {targets} but I only have temporary AWS credentials which I can generate on-demand with a simple API request. The problem is that these temporary credentials are valid for just one hour, so if my targets pipeline takes more than an hour, targets would not be able to access the S3 bucket once the initial credentials expire.

Any thoughts on possible approaches/solutions?

[wlandau]

That's a tough one. Probably requires a conversation with your sys admin. There is precedent though: S3 usage in Nextflow is such a standard pattern these days for non-targets pipelines. So it's not unreasonable to ask for a longer lifetime for credentials, or an alternative computing environment where those credentials persist. The more I listen to conversations about cloud security, the more it seems to be about "least privilege", not about taking away all privileges outright. Admins just need to know that longer credentials is an actual need and not something you can work around.

[solmos]

I was afraid there would not be a solution. I will try to ask for longer credential lifetime... Thanks for the help!

[noamross]

If I recall @cboettig created https://github.com/boettiger-lab/earthdatalogin/ to tackle a very similar problem of only temporary credentials being available for NASA data stored at AWS. The package itself has some functions to automate this credential getting and setting (https://github.com/boettiger-lab/earthdatalogin/blob/main/R/edl_s3_token.R). I think he had originally set it up as a background process so the environment variables would be changed regularly for out-of-R tools like GDAL to work properly. This is mostly a hack, and one would have to think about how to propagate updated environment variables to the various targets subprocesses. (The underlying paws library will read from .aws/credentials if environment variables are not available, for instance). Carl might have some additional advice, though.

[wlandau]

A separate background process to rotate .aws/credentials would be nice because then targets would not have to rotate environment variables for a running session. That would take care of the local session. I wonder what it would take for AWS Batch workers (e.g. from crew.aws.batch) to pick up the rotated credentials, or if the credentials are unnecessary in a Batch job because of the IAM policy.

I am not sure exactly how credentials and IAM roles are dispatched, but I suspect the credential rotation mechanics may vary from organization to organization. Another reason to consider a solution that can run separately alongside a targets pipeline.