Host function `generic_hash()` with blake2b/blake3 support. #4412

koxu1996 · 2023-11-13T17:33:55Z

In order to support Zero Knowledge software, cryptography related utils should be implemented as a host functions, otherwise execution cost would be too high. For example running Risc0 VM proof verification for simple sum of squares takes more than 6000 CSPR. The underlying hash used: sha256.

This PR adds host function called generic_hash(input, type) with support for the following types:

HashAlgoType::Blake2b - used existing blake2b implementation,
HashAlgoType::Blake3b - blake3 library was introduced.

We expect to add more algorithms, i.e. sha256, keccak, poseidon.

Example usage

Smart contract:

#![no_std]
#![no_main]

extern crate alloc;

use alloc::format;
use casper_contract::contract_api::{crypto, runtime};
use casper_types::HashAlgoType;

#[no_mangle]
pub extern "C" fn call() {
    let input = "casper".as_bytes();
    runtime::print(&format!("Input: {:?}", input));

    let blake2b = crypto::generic_hash(input, HashAlgoType::Blake2b);
    runtime::print(&format!("Blake2b hash: {:?}", blake2b));

    let blake3 = crypto::generic_hash(input, HashAlgoType::Blake3);
    runtime::print(&format!("Blake3 hash: {:?}", blake3));
}

Node output:

Input: [99, 97, 115, 112, 101, 114]
Blake2b hash: [163, 40, 63, 203, 149, 170, 125, 252, 231, 127, 90, 198, 44, 250, 234, 42, 1, 199, 20, 158, 150, 188, 25, 205, 21, 106, 255, 40, 204, 65, 45, 196]
Blake3 hash: [122, 158, 116, 51, 175, 213, 20, 50, 237, 63, 251, 230, 53, 2, 233, 86, 36, 159, 195, 109, 254, 108, 189, 210, 233, 1, 93, 219, 226, 1, 50, 217]

Considerations

~~1. AssemblyScript support~~

~~Is AssemblyScript binding required? I am pretty sure no one uses it.~~

Update: AssemblyScript is broken since a long time. I see no reason to include bindings for something that does not work.

~~2. Gas cost~~

For now, I used the same cost for generic_hash() as defined for blake2() - default fixed value 200. I can do benchmarks for WASM bytecode execution, but is there any conversion ratio between CPU time and execution cost?

Update: There will be new execution engine that will affect (lower) gas usage. It will be the better to make benchmarks then.

3. Place for core hashes implementation

Blake2b is implemented in types package, which does not seem to be reasonable. I added blake3 to new module crypto in execution engine, but I am open to discussing its location.

4. Single vs multiple host function

Do we want to introduce separate host functions for each hash algorithm?

5. Place for generic_hash() in contract API

I was thinking of using runtime, but finally decided to create new crypto module.

Will be used to introduce hashing method for runtime.

Placed in new `crypto` module under `runtime`.

Will be used to switch between different hashing algorithms.

Note: no gas charging was introduced yet.

Inspired by `FunctionIndex::Blake2b`, which is also not researched in terms of execution cost.

Note: new module `crypto` was introduced.

execution_engine/Cargo.toml

execution_engine/src/core/runtime/externals.rs

smart_contracts/contract/src/contract_api/crypto.rs

mpapierski · 2023-11-14T13:04:19Z

types/src/crypto.rs

+    Blake3 = 1,
+}
+
+impl ToBytes for HashAlgoType {


You can skip ToBytes/FromBytes for this type as it's easier to pass it by value. Same with impl AsRef I think.

I removed both, and introduced TryFrom<u8> - see 2c70942.

Btw. handling error was crazy, as compiler could not determine that casper_types::bytesrepr::Error should be converted into casper_execution_engine::core::execution::error, and then into casper_wasmi_core::trap::Trap. Finally I found working line:

let hash_algo_type = HashAlgoType::try_from(hash_algo_type as u8).map_err(|e| Error::from(e))?;

types/src/crypto.rs

Results in better performance. See https://github.com/BLAKE3-team/BLAKE3/blob/92e4cd71be48fdf9a79e88ef37b8f415ec5ac210/Cargo.toml#L21-L26 Co-authored-by: Michał Papierski <[email protected]>

Improved type documentation. Co-authored-by: Michał Papierski <[email protected]>

We can avoid working with bytes and serialization, as `HashAlgoType` enum is represented by simple `u8`.

After passing `hash_algo_type` directly, we have one less parameter.

mpapierski

Two remaining comments and looks good otherwise 👍

mpapierski · 2023-11-16T11:16:30Z

types/src/crypto.rs

+    type Error = bytesrepr::Error;
+
+    fn try_from(value: u8) -> Result<Self, bytesrepr::Error> {
+        FromPrimitive::from_u8(value).ok_or(bytesrepr::Error::Formatting)


I'd probably use ApiError::InvalidArgument here to make it more obvious, since Formatting error may be confusing as there's no serialization involved

I agree Formatting error might be confusing. Why I did that? I based on ExecutionResultTag and few other enums in execution_result.rs, but now I see those TryFrom's are always used internally inside ExecutionResult::from_bytes().

I think ApiError should not be present in base Casper types, as no API is involved here. Therefore I think we can just keep it simple and make () the error - 85885d0:

We can return ApiError::InvalidArgument inside function handler - it makes sense - but it complicates the code a little bit. Nevertheless, done in d064e2c.

Additional, please take in mind that user will not able to face this error, because using HashAlgoType enum as an argument forces the correct value.

mpapierski · 2023-11-16T11:19:36Z

execution_engine/src/core/runtime/externals.rs

+                        }
+                    })?;
+
+                let result = if digest.len() > out_size as usize {


Something I just noticed - we've done hashing already, but we're checking the size after the fact. Since the output of the hash is constant, we could also move this check above the checked_memory_slice line.

Even though the output is always expected to be 32 bytes, those hashes use different constants for their digest size:

blake2: BLAKE2B_DIGEST_LENGTH

blake3: BLAKE3_DIGEST_LENGTH

Unless we implement all the necessary hash functions (e.g., sha256, keccak, poseidon) and generalize it with something like GENERIC_HASH_DIGEST_LENGTH, I would prefer to avoid using hardcoded pre-checks.

It can be simple `()` instead of `bytesrepr::Error::Formatting` (the one commonly used for serialization related issues).

Previous deserializiation error was confusing. NOTE: The code seems too complex for such a basic thing like returning API error.

koxu1996 · 2023-11-20T15:27:29Z

I just confirmed that host function is still working correctly after all changes and fixes.

Detected with `cargo fmt`.

EdHastingsCasperAssociation · 2024-02-27T18:09:20Z

Michal's approval is sufficient for you to move forward with this at the functional level. As far as inclusion into 1.6, @marc-casperlabs is the principal engineer on that rel; I've had no direct involvement with it.

If it can't go into 1.6 for timeline reasons, we can repoint it to 2.0 and then I'll review it.

EdHastingsCasperAssociation · 2024-04-23T16:16:15Z

You should repoint this to feat-2.0.

EdHastingsCasperAssociation · 2024-09-05T18:24:51Z

@mpapierski should this be repointed to feat-2.0 ?

mpapierski · 2024-09-06T09:24:48Z

@mpapierski should this be repointed to feat-2.0 ?

I talked to @asladeofgreen and @jonas089 and for now, there's no concrete use case for generic hash functions, but we can revisit this once we have use for this internally or externally from the community. My understanding is that this PR was done as part of research.

devendran-m · 2024-09-19T09:05:45Z

Related issue (created to take this forward) #4892

koxu1996 added 8 commits November 13, 2023 18:30

Install blake3 package in execution_engine.

2dc1b1a

Will be used to introduce hashing method for runtime.

Add blake3 hasher to execution engine.

930d0b8

Placed in new `crypto` module under `runtime`.

Add HashAlgoType to base types.

356b5ca

Will be used to switch between different hashing algorithms.

Implement host function GenericHash.

eb3eb45

Note: no gas charging was introduced yet.

Charge default cost for using generic_hash.

267873a

Inspired by `FunctionIndex::Blake2b`, which is also not researched in terms of execution cost.

Introduce casper_generic_hash FFI in contract.

f27876d

Add crypto::generic_hash() to contract API.

ef78d59

Note: new module `crypto` was introduced.

Update contract API changelog.

5d400ec

koxu1996 mentioned this pull request Nov 13, 2023

Host function generic_hash() with blake2b/blake3 support. #4411

Closed

mpapierski self-requested a review November 14, 2023 12:46

mpapierski reviewed Nov 14, 2023

View reviewed changes

koxu1996 and others added 3 commits November 15, 2023 18:53

Use default features for blake3 to utilize SIMD.

8bd8fbb

Results in better performance. See https://github.com/BLAKE3-team/BLAKE3/blob/92e4cd71be48fdf9a79e88ef37b8f415ec5ac210/Cargo.toml#L21-L26 Co-authored-by: Michał Papierski <[email protected]>

Update HashAlgoType description.

d38140f

Improved type documentation. Co-authored-by: Michał Papierski <[email protected]>

Pass hash_algo_type directly by value.

2c70942

We can avoid working with bytes and serialization, as `HashAlgoType` enum is represented by simple `u8`.

koxu1996 requested a review from mpapierski November 15, 2023 20:22

Fix signature (arguments length) in WASM resolver.

4ec1f31

After passing `hash_algo_type` directly, we have one less parameter.

mpapierski approved these changes Nov 16, 2023

View reviewed changes

koxu1996 added 2 commits November 20, 2023 15:39

Change error type for HashAlgoType::try_from.

85885d0

It can be simple `()` instead of `bytesrepr::Error::Formatting` (the one commonly used for serialization related issues).

Return ApiError::InvalidArgument for invalid hash_algo_type.

d064e2c

Previous deserializiation error was confusing. NOTE: The code seems too complex for such a basic thing like returning API error.

ghost requested a review from EdHastingsCasperAssociation November 20, 2023 15:34

Fix formatting.

4b85568

Detected with `cargo fmt`.

EdHastingsCasperAssociation requested review from marc-casperlabs and removed request for EdHastingsCasperAssociation February 27, 2024 18:05

mpapierski closed this Sep 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Host function `generic_hash()` with blake2b/blake3 support. #4412

Host function `generic_hash()` with blake2b/blake3 support. #4412

koxu1996 commented Nov 13, 2023 •

edited

Loading

mpapierski Nov 14, 2023

koxu1996 Nov 15, 2023

mpapierski left a comment

mpapierski Nov 16, 2023

koxu1996 Nov 20, 2023 •

edited

Loading

koxu1996 Nov 20, 2023

mpapierski Nov 16, 2023

koxu1996 Nov 20, 2023

koxu1996 commented Nov 20, 2023

EdHastingsCasperAssociation commented Feb 27, 2024 •

edited

Loading

EdHastingsCasperAssociation commented Apr 23, 2024

EdHastingsCasperAssociation commented Sep 5, 2024

mpapierski commented Sep 6, 2024 •

edited

Loading

devendran-m commented Sep 19, 2024

Host function generic_hash() with blake2b/blake3 support. #4412

Host function generic_hash() with blake2b/blake3 support. #4412

Conversation

koxu1996 commented Nov 13, 2023 • edited Loading

Example usage

Considerations

mpapierski Nov 14, 2023

Choose a reason for hiding this comment

koxu1996 Nov 15, 2023

Choose a reason for hiding this comment

mpapierski left a comment

Choose a reason for hiding this comment

mpapierski Nov 16, 2023

Choose a reason for hiding this comment

koxu1996 Nov 20, 2023 • edited Loading

Choose a reason for hiding this comment

koxu1996 Nov 20, 2023

Choose a reason for hiding this comment

mpapierski Nov 16, 2023

Choose a reason for hiding this comment

koxu1996 Nov 20, 2023

Choose a reason for hiding this comment

koxu1996 commented Nov 20, 2023

EdHastingsCasperAssociation commented Feb 27, 2024 • edited Loading

EdHastingsCasperAssociation commented Apr 23, 2024

EdHastingsCasperAssociation commented Sep 5, 2024

mpapierski commented Sep 6, 2024 • edited Loading

devendran-m commented Sep 19, 2024

Host function `generic_hash()` with blake2b/blake3 support. #4412

Host function `generic_hash()` with blake2b/blake3 support. #4412

koxu1996 commented Nov 13, 2023 •

edited

Loading

koxu1996 Nov 20, 2023 •

edited

Loading

EdHastingsCasperAssociation commented Feb 27, 2024 •

edited

Loading

mpapierski commented Sep 6, 2024 •

edited

Loading