Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: Reduce requests made to Near Lake S3 #665

Merged
merged 10 commits into from
Apr 18, 2024
Merged

fix: Reduce requests made to Near Lake S3 #665

merged 10 commits into from
Apr 18, 2024

Commits on Apr 18, 2024

  1. feat: Use shared s3 client across all Near Lake processes

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    9320fc8 View commit details
    Browse the repository at this point in the history

  2. refactor: Replace with s3_client wrapper to allow mocking

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    3cbcd5d View commit details
    Browse the repository at this point in the history

  3. test: impl Clone for S3Client wrapper

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    7dcb9df View commit details
    Browse the repository at this point in the history

  4. feat: Cache & deduplicate concurrent s3 requests

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    63b2c44 View commit details
    Browse the repository at this point in the history

  5. test: Add tests for SharedLakeS3Client

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    02de1a0 View commit details
    Browse the repository at this point in the history

  6. fix: Ensure list_objects() lists more than 1 object

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    b8e058c View commit details
    Browse the repository at this point in the history

  7. refactor: Cache fixed amount of s3 futures

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    cc8e62c View commit details
    Browse the repository at this point in the history

  8. feat: Measure lock wait time in LakeS3Client

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    55f60a6 View commit details
    Browse the repository at this point in the history

  9. feat: Create separete s3 prefix/start_after methods

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    0f4fea5 View commit details
    Browse the repository at this point in the history

  10. test: Make lake_s3_client tests less flaky (hopefully)

    @morgsmccauley
    morgsmccauley committed Apr 18, 2024
    Configuration menu
    Copy the full SHA
    e2eba38 View commit details
    Browse the repository at this point in the history