Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: full memtrie logic for range retain #12130

Merged
merged 4 commits into from
Sep 24, 2024
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 2 additions & 32 deletions core/store/src/trie/mem/loading.rs
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,7 @@ mod tests {
};
use crate::trie::mem::loading::load_trie_from_flat_state;
use crate::trie::mem::lookup::memtrie_lookup;
use crate::trie::mem::nibbles_utils::{all_nibbles, full_16ary_tree_keys};
use crate::{DBCol, KeyLookupMode, NibbleSlice, ShardTries, Store, Trie, TrieUpdate};
use near_primitives::congestion_info::CongestionInfo;
use near_primitives::hash::CryptoHash;
Expand Down Expand Up @@ -300,18 +301,6 @@ mod tests {
check_maybe_parallelize(keys, true);
}

fn nibbles(hex: &str) -> Vec<u8> {
if hex == "_" {
return vec![];
}
assert!(hex.len() % 2 == 0);
hex::decode(hex).unwrap()
}

fn all_nibbles(hexes: &str) -> Vec<Vec<u8>> {
hexes.split_whitespace().map(|x| nibbles(x)).collect()
}

#[test]
fn test_memtrie_empty() {
check(vec![]);
Expand Down Expand Up @@ -354,26 +343,7 @@ mod tests {

#[test]
fn test_full_16ary_trees() {
check(all_nibbles(
"
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f
40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f
60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f
70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f
a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af
b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf
c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf
d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df
e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef
f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff
",
))
check(full_16ary_tree_keys())
}

#[test]
Expand Down
2 changes: 2 additions & 0 deletions core/store/src/trie/mem/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ pub mod loading;
mod lookup;
pub mod mem_tries;
pub mod metrics;
#[cfg(test)]
pub(crate) mod nibbles_utils;
pub mod node;
mod parallel_loader;
pub mod resharding;
Expand Down
34 changes: 34 additions & 0 deletions core/store/src/trie/mem/nibbles_utils.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
pub(crate) fn hex_to_nibbles(hex: &str) -> Vec<u8> {
if hex == "_" {
return vec![];
}
assert!(hex.len() % 2 == 0);
hex::decode(hex).unwrap()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that right? Hex decodes to bytes but nibbles are half-bytes. I do realize you only moved it and it's probably ok so I'm mostly curious where is the gap in my understanding.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the Robin's assumption was that we can only write actual bytes and we can't write key with odd number of nibbles. Maybe trie internals shouldn't be aware of that; on the other hand, vectors of even lengths cover all cases well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was expecting the bytes to be further split down into nibbles, so something like this:

hex::decode(hex).unwrap().iter().map(|byte| vec![byte & 0xf0 >> 4, byte & 0x0f]).collect_vec().concat()

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, each string symbol represents a readable nibble (0..9, a..f). I'll add comment

}

pub(crate) fn all_nibbles(hexes: &str) -> Vec<Vec<u8>> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: how about something like multi_hex_to_nibbles?

hexes.split_whitespace().map(|x| hex_to_nibbles(x)).collect()
}

pub(crate) fn full_16ary_tree_keys() -> Vec<Vec<u8>> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: How about all_two_nibble_nibbles or all_eight_bit_nibbles?

all_nibbles(
"
00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f
10 11 12 13 14 15 16 17 18 19 1a 1b 1c 1d 1e 1f
20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e 2f
30 31 32 33 34 35 36 37 38 39 3a 3b 3c 3d 3e 3f
40 41 42 43 44 45 46 47 48 49 4a 4b 4c 4d 4e 4f
50 51 52 53 54 55 56 57 58 59 5a 5b 5c 5d 5e 5f
60 61 62 63 64 65 66 67 68 69 6a 6b 6c 6d 6e 6f
70 71 72 73 74 75 76 77 78 79 7a 7b 7c 7d 7e 7f
80 81 82 83 84 85 86 87 88 89 8a 8b 8c 8d 8e 8f
90 91 92 93 94 95 96 97 98 99 9a 9b 9c 9d 9e 9f
a0 a1 a2 a3 a4 a5 a6 a7 a8 a9 aa ab ac ad ae af
b0 b1 b2 b3 b4 b5 b6 b7 b8 b9 ba bb bc bd be bf
c0 c1 c2 c3 c4 c5 c6 c7 c8 c9 ca cb cc cd ce cf
d0 d1 d2 d3 d4 d5 d6 d7 d8 d9 da db dc dd de df
e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 ea eb ec ed ee ef
f0 f1 f2 f3 f4 f5 f6 f7 f8 f9 fa fb fc fd fe ff
",
)
}
Loading
Loading