-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nested Store Prefix Iterators Interfere With Each Other #14786
Comments
So I'm pretty sure nested iterators are not supported by IAVL/tm-db in the SDK. I'm not sure if there's explicit documentation of this? I do know ICS used nested iterators in various places which causes many issues, once they were removed, the issues resolved. I'm also pretty confident you won't find any examples of them in the SDK's core modules, but I could be wrong. That being said, it seems like a case we should support. |
I couldn't find anything specific in documentation either. It's something that worked just fine on v0.46.7 (and before), though. |
In our cosmos fork, I reverted that PR and pulled the result into the It's probably worth noting here that during the run of both iterators discussed in the bug report, there are only store reads being done (no store writes). |
there was a testcase added that fixed the issue present before, there were also more reports of the same issue. We should aim to keep that test passing and see how to modify to have tests pass that replicate the provanence design |
Are all of these keys used with the same store? Your code uses the same StoreKey, so it appears so? Unless your module ensures that the first bytes of each of those type bytes never overlap with any other type byte definitions, this doesn't work. There's no way for iterators, or prefix stores, to distinguish between keys with the same prefix. |
Probably not related to this issue, I just find that you may need to do |
func TestIteratorNested(t *testing.T) {
mem := dbadapter.Store{DB: dbm.NewMemDB()}
store := cachekv.NewStore(mem)
// setup:
// - (owner, contract id) -> contract
// - (contract id, record id) -> record
owner1 := 1
contract1 := 1
contract2 := 2
record1 := 1
record2 := 2
store.Set([]byte(fmt.Sprintf("c%04d%04d", owner1, contract1)), []byte("contract1"))
store.Set([]byte(fmt.Sprintf("c%04d%04d", owner1, contract2)), []byte("contract2"))
store.Set([]byte(fmt.Sprintf("r%04d%04d", contract1, record1)), []byte("contract1-record1"))
store.Set([]byte(fmt.Sprintf("r%04d%04d", contract1, record2)), []byte("contract1-record2"))
store.Set([]byte(fmt.Sprintf("r%04d%04d", contract2, record1)), []byte("contract2-record1"))
store.Set([]byte(fmt.Sprintf("r%04d%04d", contract2, record2)), []byte("contract2-record2"))
it := types.KVStorePrefixIterator(store, []byte(fmt.Sprintf("c%04d", owner1)))
defer it.Close()
var records []string
for ; it.Valid(); it.Next() {
contractID, err := strconv.ParseInt(string(it.Key()[5:]), 10, 32)
require.NoError(t, err)
it2 := types.KVStorePrefixIterator(store, []byte(fmt.Sprintf("r%04d", contractID)))
for ; it2.Valid(); it2.Next() {
records = append(records, string(it2.Value()))
}
it2.Close()
}
require.Equal(t, []string{
"contract1-record1",
"contract1-record2",
"contract2-record1",
"contract2-record2",
}, records)
} I wrote a similar test case according to the description, but it succeeds, is there difference in the case, or ideally can you write a test case that reproduce in sdk? @SpicyLemon |
Yes. The number in parentheses there is the int value used for each type. I.e. |
Yeah. Most of our keys stuff should clone and append instead of |
Yeah. I'll see if I can. Maybe there's something about using a different instance of the |
Oh! Sorry, I thought the numbers meant byte width, not byte value. |
After diving into rabbit holes for a few hours, I finally got a test to fail:
The only difference between this one and the one you shared earlier, is that I switched the Before making that switch, I looked at the debugger and saw that the record entries were still being added to the index iterator. The reason it wasn't a problem is that they were greater than the iterator's end key. Here's what the failure looks like:
It could probably be tweaked/clarified to assert that the index iterator ( The first time through, |
I added that test to three branches:
Feel free to update/use those branches for whatever. |
I tweaked the test a bit (on each branch) to keep track of the keys that the index iterator ( The failure message now looks like this:
|
Closes: cosmos#14786 Solution: - do a copy on btree before iteration
fixed, thanks for your priceless test case, @SpicyLemon |
Thank you to both of you for the tremendous help here!!! |
@alexanderbez It does look painfully familiar :) It's hard to tell if it's the same problem as we had nested iterators all over our implementation. We just decided to avoid doing that. Now we are much more conservative with our use of iterators. |
I don't see any PRs associated with this. How was it completed? |
|
I thought the pr was linked. Sorry for not sending it your way. Let us know if you have any feedback and we can incorporate |
Yay for Github! My best guess is that Github's auto-linking/closing parsing stuff is case sensitive, looking for all-lowercase "closes" (ignoring "Closes"). All good now though! Thanks! |
Summary of Bug
Items from a nested store prefix iterator are added to other existing (still open and in use) store prefix iterators. This causes these other iterators to provide entries outside their intended range.
Version
v0.46.8
Steps to Reproduce
I found this because one of our unit tests started failing when we bumped our SDK to version
v0.46.8
. I'll try to describe it as succinctly as possible, but will also provide code at the bottom.The function being testing named
IterateRecordSpecsForOwner
. It iterates over some index entries in state in order to get some specific "contract specification ids"; I will refer to this as the "outside iterator". For each contract specification id, it then iterates over the record specification's in state; I will refer to these ones as the "inside iterator(s)".The iterators are created using
sdk.KVStorePrefixIterator
. The inside iterators are created, fully used, and closed while the outside iterator is still open and being used, i.e. the iterators are nested. The same store is provided for both, but different prefixes are provided that don't overlap.There are three types of state entries involved in this unit test:
<type byte (32)><address length><address><contract specification id>
.<type byte (3)><contract spec id>
. These aren't directly involved in the iteration, but are needed as part of setup.<type byte (5)><contract spec id><record spec name hash>
.The outer iterator is prefixed with
<type byte (32)><address length><address>
. It's.Key()
s are expected to be<contract specification id>
s.The inside iterator is prefixed with
<type byte (5)><contract spec id>
. The entries it provides are expected to be record specifications.The unit test that's failing has the following set up:
After setup,
IterateRecordSpecsForOwner
is called for account 1. It is expected to iterate over four record specs: two each from contract specs 0 and 2. Instead, the second value provided by the outer iterator is a record spec entry (which it fails to decode since it's expecting only index entries).In IntelliJ, i used the debugger on the unit test in question to investigate.
I verified that the store has everything expected, i.e.
store.parent.cache = {map[string]*cachekv.cValue}
has all twelve items (store = {types.KVStore | *gaskv.Store}
, andstore.parent = {types.KVStore | *cachekv.Store}
).Here's what I'm seeing as it runs:
.Valid()
called on outside iterator: returns true..Key()
called on outside iterator: returns the first index key as expected (account 1 -> contract spec 0).Here's what the debugger had to say about the outside iterator:
it.parent.parent = {db.Iterator | *iavl.UnsavedFastIterator}
.Valid()
isfalse
, butit.parent.cache
.Valid()
istrue
.Also not included is that,
it.parent.cache.iter.stack[0]
:.i
=0
and.n.items
has 2 entries: the two index entries expected..Next()
called on outside iterator..Valid()
called on outside iterator: returns true..Key()
called on outside iterator: returns the key for the second record specification that was seen in step 5.Here's what the debugger had to say about the outside iterator now:
it.parent.parent
.Valid()
isfalse
andit.parent.cache
.Valid()
istrue
.Also not included in that,
it.parent.cache.iter.stack[0]
:.i
= 1,.n.items
has 4 entries: the two record specs seen in step 5, then the two index entries.The
it.parent.cache.iter.tr.root.items
entries are[0]
= the first record spec seen by the inside iterator,[1]
= the second record spec seen,[2]
= the first index entry seen,[3]
= the original second index entry (this is what the iterator is expected to be on right now).At this point,
store.parent.unsortedCache = {map[string]struct{}}
has eight items, andstore.parent.sortedCache = {*internal.BTree | 0x14000fec4d0}
has the four items that the outside iterator now has..Key()
from the outside iterator.Code
The unit test: TestIterateRecordSpecsForOwner
All assertions in that test fail, the first of which is on line 396.
IterateRecordSpecsForOwner:
IterateContractSpecsForOwner:
IterateRecordSpecsForContractSpec:
The text was updated successfully, but these errors were encountered: