Prod Release 15/02/24 #561
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The start block parameter (`start_block_height`) has two states, signifying "Start from block" and "Start from latest" only. With the removal of the continuous "real-time" process, which essentially runs Indexers "From Interruption", the need for a third state arises, representing "Interruption" arises. This PR expands the registry contract/types to accommodate this change. This work was a good opportunity to refactor the existing types, removing unnecessary fields and tailoring it to the new architecture. ## Registry Types Refactoring - `filter: IndexerRule` -> `rule: Rule`: `IndexerRule` contained a lot of noise/data which was not needed. `id`, `name`, and `indexer_rule_kind` have all been removed. The only useful part here is `MatchingRule` which has been renamed to just `Rule`. - `schema` is now non-optional: This field is always required so it makes sense to convey that in the code. Having it as an `Option` created lots of unnecessary checks throughout the system. - `start_block_height` replaced by `start_block`: The latter being an `enum` to represent "From Latest", "From Block", and "From Interruption" ## Public methods/API To minimise the disruption of the existing contract consumers, the public API remains unchanged, i.e. the core methods (`list_indexer_functions`, `register_indexer_function`, etc.) use/return the same types. As the underlying data will be migrated to the new types, these methods infer the old types from the new ones. New methods have been created to work with the new types (`register`, `list_all`, etc.) allowing clients to move over when possible, as opposed to creating a breaking change.
feat: Add Metrics for Memory Statistics
When an Executor is stopped, the metrics exposed from that worker still persist. As metrics are aggregated across all workers, the metrics for a given indexer would be aggregated across all previous workers, creating incorrect metrics.
`xread` returns messages _after_ the provided ID, there's no need to increment the ID ourselves. If we increment to an ID that actually exists within the stream, the message will be skipped permanently. e.g. we have messages `1-0` and `1-1`, on first read we get `1-0`, we increment to `1-1`, then on the second read we get nothing.
…#553) This PR adds support for the updated Registry types, including the new `StartBlock` config option, across the Block Streamer/Coordinator. With `StartBlock`, the logic within Coordinator is much more straight forward - we no longer need to guess whether we should "continue" or "start over", that is build in to the configuration options. This only really affects the handling of Block Streams, the executor flow remains the same: always restart when a new version is published. To summarise the Block Stream synchronisation process: - `StartBlock::Continue` - Resumes process, keeping the current data in the Redis Stream, and start the Block Stream from `last_published_block` - `StartBlock::Latest` - Starts a new process, clears the Redis Stream, starts the Block Stream from the registry version, essentially being latest - `StartBlock::Height(u64)` - Starts a new process, clears the Redis Stream, starts the Block Stream from the height configured Additionally, Accounts/Indexers which have just been migrated, and also streams which have been stopped but have unchanged versions (e.g. after Block Streamer restart), will be treated the same as `StartBlock::Continue`.
A line to throw the error was mistakenly added during testing of things. This should not be thrown since it prevents retries.
darunrs
approved these changes
Feb 15, 2024
| storage::get(redis_connection_manager, storage::DENYLIST_KEY).await?; | ||
| let raw_denylist: String = storage::get(redis_connection_manager, storage::DENYLIST_KEY) | ||
| .await | ||
| .unwrap_or("".to_owned()); |
There was a problem hiding this comment.
This should be returning "[]" in order to not cause problems in fact. I can either push a quick PR to merge that change or we can let it through, since it isn't a problem if there's a denylist anyway (You can add it in redis before merging).
There was a problem hiding this comment.
[] has already been added to Redis :)
| #[derive(BorshDeserialize, BorshSerialize, Debug)] | ||
| pub struct Contract { | ||
| registry: IndexersByAccount, | ||
| pub struct OldContract { |
There was a problem hiding this comment.
Have you deployed the updates to prod's registry already?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
StartBlockoptions within Coordinator & Block Streamer (feat: HandleStartBlockoptions within Coordinator & Block Streamer #553)