Prod Release 16/07/24 #869
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replaces Delta Lake Stream with Bitmap Indexer Stream as the method for backfilling matching block heights before switching to real time processing. Bitmap Indexer resolves many issues the Delta Lake approach had: 1. Can be extended more easily. 2. Can be queried much faster thanks to storage in a database as well as in a compressed format. 3. Queries can be customized, enabling better matching. 4. Supports wildcard contract filters much better than the previous approach. 5. Is much harder to overload with too much data, which was a problem with the previous approach. --------- Co-authored-by: Morgan Mccauley <[email protected]>
`near-lake-framework` will infinitely retry, making our tests infinitely hang. This PR monitors the `last_processed_ block`, and cancels the task from the outside when the last desired block is reached.
Both the `IndexerState` and `IndexerConfig` structs are used individually to access Redis, and therefore have duplicate code for generating the required keys. This is problematic for the following reasons: 1. Redis access is limited to these structs, anything else requiring access needs to duplicate the logic again 2. Updates need to be reflected across both structs, creating more room for error This PR creates a common Redis `KeyProvider` interface/trait for accessing Indexer Redis data/keys. Now the required structs, i.e. `IndexerState` and `IndexerConfig`, can implement this trait, and automatically get access to the methods which generate the keys. Also did some minor clean up, removing the old Coordinator migration code.
…857) Cause of multiple rerendering on page load
Optimized Editor Lifecycle methods and removed the 'janky' Alert that would insert itself between the developer tools and Editor component in favor of a small icon that is in the fileswitcher.
As we fetch blocks for a given day, there is possibility for that day to include blocks lesser than the provided `start_block_height`. This PR ensures that those blocks are filtered out.
In an effort to understand the bitmap implementation, I've added documentation comments. Let me know if it's incorrect! --------- Co-authored-by: Darun Seethammagari <[email protected]>
Context DB in the frontend was showing an error for inserts and select. For inserts, some types such as `date` would expect `Date` type when string type also works. I updated the type gen to allow string type input for these options. This ensures string or number input is acceptable for essentially all types. <img width="955" alt="image" src="https://github.com/near/queryapi/assets/22734869/3dd9ae4b-fa95-44de-824e-b026c525c5d1"> In addition, selection queries allow for array input to match muliple values for a certain column. However, the frontend was not updated to accept this input. I've added a new interface called Query which allows for array values in input objects.
…r card in the dashboard (#837) BOS component indexercard now queries past 7 day query count.
Option is more convenient as we don't need to manually inspect the error to determine whether or not the Indexer exists. This is a breaking change, but there aren't any consumers of this so it is safe.
If the stream yielding block heights from the bitmap indexer fails, its error will be eaten by the while loop as the loop will simply end if any error takes place. This PR ensures that if the result from the stream is an error, it logs the error before proceeding with Lake.
…r metadata if available in databricks (#873) BOS component data is fetched from NEAR RPC call 'list_all'. We use a map to set a k/v pair for indexer metadata recieved on databricks. When we render each card we check the map to see if the indexer to be rendered from NEAR RPC exist in the map. If it does exist we conditionally render metadata.
darunrs
approved these changes
Jul 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
bitmapimplementation (doc: Documentbitmapimplementation #862)