Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: Speed up Historical Backfill process #379

Merged
merged 10 commits into from
Nov 12, 2023
Merged

refactor: Speed up Historical Backfill process #379

merged 10 commits into from
Nov 12, 2023

Commits on Nov 9, 2023

  1. chore: Log on historical backfill completion

    @morgsmccauley
    morgsmccauley committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    6e3d0a6 View commit details
    Browse the repository at this point in the history

  2. feat: Reset Redis state prior to historical backfill

    @morgsmccauley
    morgsmccauley committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    58a34f7 View commit details
    Browse the repository at this point in the history

  3. feat: Flush blocks from index before continuing historical backfill

    @morgsmccauley
    morgsmccauley committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    fdd08ed View commit details
    Browse the repository at this point in the history

  4. refactor: Configure AWS/Lake through environment

    @morgsmccauley
    morgsmccauley committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    882da05 View commit details
    Browse the repository at this point in the history

  5. chore: Update historical backfill logs

    @morgsmccauley
    morgsmccauley committed Nov 9, 2023
    Configuration menu
    Copy the full SHA
    745ff21 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2023

  1. refactor: Use Lake Framework to process unindexed historical blocks

    @morgsmccauley
    morgsmccauley committed Nov 10, 2023
    Configuration menu
    Copy the full SHA
    53649ff View commit details
    Browse the repository at this point in the history

Commits on Nov 12, 2023

  1. chore: Update historical log messages

    @morgsmccauley
    morgsmccauley committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    87b09c4 View commit details
    Browse the repository at this point in the history

  2. refactor: Move S3 related const to s3 module

    @morgsmccauley
    morgsmccauley committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    1e6c24d View commit details
    Browse the repository at this point in the history

  3. fix: Ensure unfiltered process starts where databricks finished

    @morgsmccauley
    morgsmccauley committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    0bcd8d7 View commit details
    Browse the repository at this point in the history

  4. chore: Log when filtering large amounts of unindexed blocks

    @morgsmccauley
    morgsmccauley committed Nov 12, 2023
    Configuration menu
    Copy the full SHA
    352c7e0 View commit details
    Browse the repository at this point in the history