Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'readinfo' shim to botocore.response.StreamingBody #3108

Open
1 of 2 tasks
jimdigriz opened this issue Jan 29, 2024 · 1 comment
Open
1 of 2 tasks

Add 'readinfo' shim to botocore.response.StreamingBody #3108

jimdigriz opened this issue Jan 29, 2024 · 1 comment
Labels
feature-request This issue requests a feature. needs-review This issue or pull request needs review from a core team member. p2 This is a standard priority issue

Comments

@jimdigriz
Copy link

Describe the feature

Provide a readinto method for botocore.response.StreamingBody.

Use Case

Since Python 3.11 we now have hashlib.file_digest() which can take a fileobj-ish object and return the digest, this is useful to strap to the back of .get()['Body'] without having to do the slurping of the stream yourself.

This does not work as hashlib.py:file_digest() tests for readinto and finds it missing, so raises a ValueError exception.

Proposed Solution

This is an example of shimming it into place and it Works For Me(tm):

def checksum(key):

    def _readinto(self, buf):

        return self._raw_stream.readinto(buf)

    s3 = s3_resource()
    object = s3.Object(key=key)

    if object.checksum_sha256:
        return b64decode(object.checksum_sha256).hex()

    response = object.get()
    setattr(response['Body'].__class__, 'readinto', _readinto)

    checksum = hashlib.file_digest(response['Body'], 'sha256')

    return checksum.hexdigest()

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

SDK version used

1.34.29

Environment details (OS name and version, etc.)

Debian GNU/Linux 12 (bookworm)

@jimdigriz jimdigriz added feature-request This issue requests a feature. needs-triage This issue or PR still needs to be triaged. labels Jan 29, 2024
@tim-finnigan tim-finnigan self-assigned this May 8, 2024
@tim-finnigan tim-finnigan added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label May 8, 2024
@tim-finnigan
Copy link
Contributor

Thanks for the feature request, I'll mark this issue for further review by the team. Please let us know if there are any further details you can share regarding your use case and any alternatives you have tried.

@tim-finnigan tim-finnigan added needs-review This issue or pull request needs review from a core team member. p2 This is a standard priority issue and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. needs-triage This issue or PR still needs to be triaged. labels May 8, 2024
@tim-finnigan tim-finnigan removed their assignment May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request This issue requests a feature. needs-review This issue or pull request needs review from a core team member. p2 This is a standard priority issue
Projects
None yet
Development

No branches or pull requests

2 participants