Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize merkle hashing performance in hash_tree_root #93

Closed
ralexstokes opened this issue Jul 15, 2023 · 3 comments
Closed

Optimize merkle hashing performance in hash_tree_root #93

ralexstokes opened this issue Jul 15, 2023 · 3 comments
Labels
enhancement New feature or request

Comments

@ralexstokes
Copy link
Owner

ralexstokes commented Jul 15, 2023

A big bottleneck of consumers of this crate is the performance of computing large Merkle trees. There are sophisticated caching techniques we can use to mitigate this; see #17. Separately, we can optimize the hashing itself, as computing the parent nodes of a set of child nodes in one layer of the Merkle tree can be done in parallel. This issue aims to address this latter performance technique.

relevant repo for investigating hashing techniques we may be able to use:

https://github.com/OffchainLabs/sszpp

@ralexstokes
Copy link
Owner Author

from convo's with @potuz

the entry point for your spare tree hash function
https://github.com/ralexstokes/ssz-rs/blob/main/ssz-rs/src/merkleization/mod.rs#L123
And this is the key point that you use this signature for hashing:
https://github.com/ralexstokes/ssz-rs/blob/main/ssz-rs/src/merkleization/mod.rs#L210

So your hasher function takes as input 2 chunks which are then combined into a 64 byte block and then hashed. If you could adapt this routine to use a hasher that takes a slice of bytes, multiple of 64 as input, and another preallocated buffer, multiple of 32 as output, so that the hasher hashes continously all 64 bytes chunks and spits out the consecutive 32 byte hashes in the output slice, then you can trivially switch to the vectorized assembly without any cost. The Go version of the algorithm is here:
https://hackmd.io/80mJ75A5QeeRcrNmqcuU-g#Solving-1
And the faster version I'm using in ssz++ is here
https://github.com/OffchainLabs/sszpp/blob/main/lib/merkleize.hpp#L149

@ralexstokes
Copy link
Owner Author

latest:

optimized hashing routines here: https://github.com/prysmaticlabs/hashtree

demo: https://github.com/potuz/hashtree_rust_demo

@ralexstokes ralexstokes added the enhancement New feature or request label Mar 30, 2024
@ralexstokes ralexstokes changed the title hashing performance Hashing performance Apr 1, 2024
@ralexstokes ralexstokes changed the title Hashing performance Optimize hashing performance Apr 2, 2024
@ralexstokes ralexstokes changed the title Optimize hashing performance Optimize merkle hashing performance Apr 2, 2024
@ralexstokes ralexstokes changed the title Optimize merkle hashing performance Optimize merkle hashing performance in hash_tree_root Apr 2, 2024
@ralexstokes
Copy link
Owner Author

closing in lieu of #156

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant