Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use rasterio AWS session in RasterioSource when reading files on S3 #2197

Merged
merged 1 commit into from
Jul 26, 2024

Conversation

AdeelH
Copy link
Collaborator

@AdeelH AdeelH commented Jul 26, 2024

Overview

This PR makes RasterioSource use rasterio's AWSSession when reading from a raster file on S3, which makes it easier to read from requester-pays buckets (e.g. NAIP).

Previously, the following (ipython) snippet would not work even if the user was logged into an SSO session via the AWS CLI; the user would need to export all the credentials to the environment to make it work. This PR fixes that.

%env AWS_REQUEST_PAYER=requester

from rastervision.core.data import RasterioSource

uri = 's3://naip-analytic/pa/2019/60cm/rgbir_cog/39075/m_3907515_nw_18_060_20191019.tif'
rs = RasterioSource(uri, allow_streaming=True)
chip = rs[:100, :100]

Checklist

  • Added unit tests, if applicable
  • Updated documentation, if applicable
  • Added needs-backport label if the change should be back-ported to the previous release
  • PR has a name that won't get you publicly shamed for vagueness

Notes

This PR also makes rastervision_aws_s3 recognize the AWS_REQUEST_PAYER environment variable and allow it to override Raster Vision's AWS_S3_REQUESTER_PAYS config option.

Testing Instructions

See new unit tests.

@AdeelH AdeelH merged commit 8ab5837 into azavea:master Jul 26, 2024
2 checks passed
@AdeelH AdeelH deleted the rs_session branch July 26, 2024 17:06
Copy link

codecov bot commented Oct 24, 2024

Codecov Report

Attention: Patch coverage is 96.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 90.21%. Comparing base (a25fa34) to head (1b39b99).
Report is 39 commits behind head on master.

Files with missing lines Patch % Lines
...rvision/core/data/raster_source/rasterio_source.py 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #2197      +/-   ##
==========================================
+ Coverage   90.15%   90.21%   +0.05%     
==========================================
  Files         198      198              
  Lines        9888     9902      +14     
==========================================
+ Hits         8915     8933      +18     
+ Misses        973      969       -4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant