trie_prefetcher: alternate structure #666
Open
+328
−400
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
darioush
commented
Oct 1, 2024
darioush
commented
Oct 1, 2024
darioush
commented
Oct 1, 2024
| @@ -59,7 +59,8 @@ func filledStateDB() *StateDB { | |||
|
|
|||
| func TestCopyAndClose(t *testing.T) { | |||
| db := filledStateDB() | |||
| prefetcher := newTriePrefetcher(db.db, db.originalRoot, "", maxConcurrency) | |||
| prefetchDb := newPrefetcherDatabase(db.db, maxConcurrency) | |||
There was a problem hiding this comment.
to preserve existing UT behavior for now
darioush
force-pushed
the
trie-prefetcher-alt
branch
from
3d6f372
to
f3612cd
Compare
darioush
commented
Oct 1, 2024
darioush
commented
Oct 1, 2024
Signed-off-by: Darioush Jalali <[email protected]>
…abs/coreth into trie-prefetcher-alt
darioush
commented
Nov 18, 2024
darioush
commented
Nov 18, 2024
darioush
commented
Nov 18, 2024
darioush
commented
Nov 19, 2024
|
|
||
| // triePrefetcher is an active prefetcher, which receives accounts or storage | ||
| // items and does trie-loading of them. The goal is to get as much useful content | ||
| // into the caches as possible. | ||
| // | ||
| // Note, the prefetcher's API is not thread safe. | ||
| type triePrefetcher struct { | ||
| db Database // Database to fetch trie nodes through | ||
| db PrefetcherDB // Database to fetch trie nodes through |
There was a problem hiding this comment.
changed compared to upstream
Comment on lines
+65
to
+69
| var pdb PrefetcherDB | ||
| pdb = withPrefetcherDefaults{db} | ||
| if db, ok := db.(withPrefetcherDB); ok { | ||
| pdb = db.PrefetcherDB() | ||
| } |
There was a problem hiding this comment.
changed compared to upstream
| storageFetchers int64 | ||
| largestLoad int64 | ||
| ) | ||
| defer p.db.Close() |
There was a problem hiding this comment.
added compared to upstream
| p *triePrefetcher | ||
|
|
||
| db Database // Database to load trie nodes through | ||
| db PrefetcherDB // Database to load trie nodes through |
There was a problem hiding this comment.
modified compared to upstream
| return 0 | ||
| } | ||
| return sf.to.copies | ||
| sf.db.WaitTrie(sf.trie) |
There was a problem hiding this comment.
added compared to upstream
Comment on lines
+365
to
+367
| sf.db.PrefetchAccount(sf.trie, common.BytesToAddress(task)) | ||
| } else { | ||
| sf.db.PrefetchStorage(sf.trie, sf.addr, task) |
There was a problem hiding this comment.
modified. compared to upstream (made this more explicit to avoid overriding GetAccount and GetStorage)
|
Adding do not merge label prior to verifying performance |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why this should be merged
The current trie_prefetcher is modified significantly compared to upstream, which makes it difficult to merge changes from upstream.
This PR aims to preserve the existing behavior but only with minor inline modifications to upstream code.
I would be open to other structures with the same idea.
How this works
Uses a wrapper for DB to track the central worker pool.
Adds a couple hooks for stopping the background processes.
How this was tested
CI