feat: Implement Historical Backfill using Bitmap Indexer (#803)

Replaces Delta Lake Stream with Bitmap Indexer Stream as the method for backfilling matching block heights before switching to real time processing. Bitmap Indexer resolves many issues the Delta Lake approach had: 1. Can be extended more easily. 2. Can be queried much faster thanks to storage in a database as well as in a compressed format. 3. Queries can be customized, enabling better matching. 4. Supports wildcard contract filters much better than the previous approach. 5. Is much harder to overload with too much data, which was a problem with the previous approach. --------- Co-authored-by: Morgan Mccauley <[email protected]>
near · Jul 4, 2024 · c64f8ff · c64f8ff
1 parent a9c25c3
commit c64f8ff
Show file tree

Hide file tree

Showing 17 changed files with 14,438 additions and 7,341 deletions.
diff --git a/block-streamer/graphql/darunrs_near/get_bitmaps_exact.graphql b/block-streamer/graphql/darunrs_near/get_bitmaps_exact.graphql
diff --git a/block-streamer/graphql/darunrs_near/get_bitmaps_wildcard.graphql b/block-streamer/graphql/darunrs_near/get_bitmaps_wildcard.graphql