Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support Star Contract Filter in Block Streamer #572

Merged
merged 7 commits into from
Feb 21, 2024

Conversation

darunrs
Copy link
Collaborator

@darunrs darunrs commented Feb 20, 2024

Block Streamer will fail to process a * contract filter as it will try to fetch index files for all account IDs. This is not only extremely inefficient since we aren't planning to skip anything, but will also fail as we limit the number of accounts fetched as a sanity check.

Block Streamer, if the filter is "*" specifically, now skips fetching index files and begins reading from Lake directly instead.

This retains the behavior of block streamer reading block data before pushing the height to Redis. There is an intended migration of Runner's prefetch into Block Streamer. This change works well towards that goal. However, obviously there is double work performed here: We fetch the block, only to just put the height into Redis. If, in the short term and before we migrate prefetch from Runner to Block Streamer, we need to speed things up, we can list files form S3 instead. However, this work would be removed after the mentioned migration of work.

@darunrs darunrs marked this pull request as ready for review February 20, 2024 20:22
@darunrs darunrs requested a review from a team as a code owner February 20, 2024 20:22
@darunrs
Copy link
Collaborator Author

darunrs commented Feb 20, 2024

I did a simple manual test: I called start_stream with a modified filter of * and observed that it began pulling block heights immediately. I also verified manually that the heights were in the expected sequential order, including skipping skipped heights.

block-streamer/src/block_stream.rs Outdated Show resolved Hide resolved
block-streamer/src/block_stream.rs Outdated Show resolved Hide resolved
@darunrs darunrs linked an issue Feb 20, 2024 that may be closed by this pull request
@darunrs
Copy link
Collaborator Author

darunrs commented Feb 20, 2024

As per discussion in team meeting, I'll expand the list of things to skip delta to *, *.near, *.kaiching, *.tg to start.

@darunrs darunrs merged commit 2a5d923 into main Feb 21, 2024
4 checks passed
@darunrs darunrs deleted the support-star-backend branch February 21, 2024 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update the streamer to support this filter
2 participants