Migrations & data storage for pre-DAG-sync #831

nuttycom · 2023-05-02T16:15:30Z

Based on #857

zcash_client_backend/src/data_api.rs

zcash_client_backend/src/data_api/chain.rs

daira

Only cursory review so far.

codecov · 2023-06-14T23:32:58Z

Codecov Report

Patch coverage: 71.45% and project coverage change: +0.47 🎉

Comparison is base (fa67d9a) 69.75% compared to head (c13c8c6) 70.23%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #831      +/-   ##
==========================================
+ Coverage   69.75%   70.23%   +0.47%     
==========================================
  Files         124      127       +3     
  Lines       11618    11848     +230     
==========================================
+ Hits         8104     8321     +217     
- Misses       3514     3527      +13

Impacted Files	Coverage Δ
zcash_client_backend/src/data_api/chain/error.rs	`0.00% <0.00%> (-21.43%)`	⬇️
zcash_client_sqlite/src/error.rs	`0.00% <0.00%> (ø)`
...rc/wallet/init/migrations/add_transaction_views.rs	`61.40% <ø> (+2.08%)`	⬆️
...llet/init/migrations/received_notes_nullable_nf.rs	`71.42% <ø> (+4.76%)`	⬆️
...e/src/wallet/init/migrations/v_transactions_net.rs	`73.33% <ø> (+4.58%)`	⬆️
zcash_client_backend/src/data_api/error.rs	`8.77% <16.66%> (+0.92%)`	⬆️
zcash_client_sqlite/src/wallet/init.rs	`54.21% <25.00%> (-17.72%)`	⬇️
zcash_client_backend/src/scanning.rs	`42.23% <25.71%> (ø)`
zcash_client_sqlite/src/chain.rs	`52.52% <33.33%> (+1.04%)`	⬆️
...te/src/wallet/init/migrations/shardtree_support.rs	`53.22% <53.22%> (ø)`
... and 14 more

... and 47 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

…ning.

Local chain validation will be performed internal to `scan_cached_blocks`, and as handling of chain reorgs will need to change to support out-of-order scanning, the `validate_chain` method will be superfluous. It is removed in advance of other changes in order to avoid updating it to reflect the forthcoming changes.

nuttycom · 2023-07-02T02:01:35Z

force-pushed to address comments from code review.

In preparation for out-of-order range-based scanning, it is necessary to ensure that the size of the Sapling note commitment tree is carried along through the scan process and that stored blocks are always persisted with the updated note commitment tree size.

…scanning`

nuttycom · 2023-07-03T16:51:33Z

force-pushed to fix an off-by-one error and fix cargo fmt errors.

nuttycom · 2023-07-03T17:07:44Z

zcash_client_backend/src/data_api/chain.rs

-        },
-    )?;
+    block_source.with_blocks::<_, DbT::Error>(scan_from, limit, |block: CompactBlock| {
+        add_block_to_runner(params, block, &mut batch_runner);


It occurs to me that we should actually do the continuity check here, because there's no reason to add a block to the runner or continue the scan if the continuity check fails.

I've made this change in the forthcoming range-suggestions PR.

daira · 2023-07-03T18:46:40Z

zcash_client_backend/src/data_api.rs

    };

    pub struct MockWalletDb {
        pub network: Network,
+        pub sapling_tree: ShardTree<
+            MemoryShardStore<sapling::Node, BlockHeight>,
+            { SAPLING_SHARD_HEIGHT * 2 },


Suggested change

{ SAPLING_SHARD_HEIGHT * 2 },

{ sapling::NOTE_COMMITMENT_TREE_DEPTH },

str4d

Reviewed as of c363e71.

Cargo.toml

str4d · 2023-07-03T15:54:14Z

Cargo.toml

+[patch.crates-io]
+incrementalmerkletree = { git = "https://github.com/zcash/incrementalmerkletree.git", rev = "082109deacf8611ee7917732e19b56158bda96d5" }
+shardtree = { git = "https://github.com/zcash/incrementalmerkletree.git", rev = "082109deacf8611ee7917732e19b56158bda96d5" }
+orchard = { git = "https://github.com/zcash/orchard.git", rev = "5da41a6bbb44290e353ee4b38bcafe37ffe79ce8" }


I verified that this is the latest commit in zcash/orchard#398.

zcash_client_backend/src/data_api.rs

zcash_client_backend/src/data_api/chain.rs

zcash_client_sqlite/src/wallet.rs

str4d · 2023-07-03T21:53:22Z

zcash_client_sqlite/src/wallet.rs

+        if expected_hash != block_hash {
+            return Err(SqliteClientError::BlockConflict(block_height));
+        }


Reminding @daira to write hir comment.

zcash_client_sqlite/src/wallet/init.rs

zcash_client_backend/proto/compact_formats.proto

zcash_client_sqlite/src/wallet/init/migrations/shardtree_support.rs

daira · 2023-07-03T22:32:04Z

I would like to finish my review before this merges (which I should be able to do tonight).

Edit: had a terrible migraine. It's fine that this merged, I'll finish my review post-hoc anyway though.

str4d

utACK c13c8c6

str4d · 2023-07-04T03:35:40Z

I'm using c13c8c6 in zcash/lightwalletd#440; please do not rebase that commit.

nuttycom force-pushed the feature/pre_dag_sync branch 5 times, most recently from b9c1782 to 58412a1 Compare May 9, 2023 02:03

nuttycom force-pushed the feature/pre_dag_sync branch 8 times, most recently from 59d712d to cea2917 Compare May 24, 2023 18:47

nuttycom force-pushed the feature/pre_dag_sync branch 2 times, most recently from e7bf6e1 to 785dd37 Compare May 25, 2023 23:45

nuttycom force-pushed the feature/pre_dag_sync branch from 785dd37 to a62a2e0 Compare June 3, 2023 01:28