diff --git a/.github/workflows/markdown-linter.yml b/.github/workflows/markdown-linter.yml index 095e558818..1ef3b61695 100644 --- a/.github/workflows/markdown-linter.yml +++ b/.github/workflows/markdown-linter.yml @@ -15,7 +15,7 @@ jobs: node-version: 18 - name: Install markdownlint-cli - run: npm install -g markdownlint-cli@0.32.1 + run: npm install -g markdownlint-cli@0.39.0 shell: bash - name: Run markdownlint diff --git a/.markdownlint.yaml b/.markdownlint.yaml index 0a576746f7..aca542234b 100644 --- a/.markdownlint.yaml +++ b/.markdownlint.yaml @@ -3,3 +3,5 @@ "code_blocks": false # Disable rule for hard tabs in code blocks "MD013": false # Disable rule for line length "MD033": false # Disable rule banning inline HTML +"MD055": + "style": "leading_and_trailing" # Instead of "consistent" (default) which can vary per file, use one style across the entire project diff --git a/README.md b/README.md index b508ed9f24..2aa5e811ac 100644 --- a/README.md +++ b/README.md @@ -87,7 +87,7 @@ celestia-appd --help ### Environment variables | Variable | Explanation | Default value | Required | -| --------------- | ---------------------------------- | -------------------------------------------------------- | -------- | +|-----------------|------------------------------------|----------------------------------------------------------|----------| | `CELESTIA_HOME` | Home directory for the application | User home dir. [Ref](https://pkg.go.dev/os#UserHomeDir). | Optional | ### Create your own single node devnet @@ -119,7 +119,7 @@ This repo attempts to conform to [conventional commits](https://www.conventional ### Tools 1. Install [golangci-lint](https://golangci-lint.run/welcome/install) 1.57.0 -1. Install [markdownlint](https://github.com/DavidAnson/markdownlint) +1. Install [markdownlint](https://github.com/DavidAnson/markdownlint) 0.39.0 1. Install [hadolint](https://github.com/hadolint/hadolint) 1. Install [yamllint](https://yamllint.readthedocs.io/en/stable/quickstart.html) 1. Install [markdown-link-check](https://github.com/tcort/markdown-link-check) @@ -154,10 +154,11 @@ Package-specific READMEs aim to explain implementation details for developers th ## Audits -Date | Auditor | Version | Report ------------|-----------------------------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------------- -2023/9/15 | [Informal Systems](https://informal.systems/) | [v1.0.0-rc6](https://github.com/celestiaorg/celestia-app/releases/tag/v1.0.0-rc6) | [informal-systems.pdf](docs/audit/informal-systems.pdf) -2023/10/17 | [Binary Builders](https://binary.builders/) | [v1.0.0-rc10](https://github.com/celestiaorg/celestia-app/releases/tag/v1.0.0-rc10) | [binary-builders.pdf](docs/audit/binary-builders.pdf) +| Date | Auditor | Version | Report | +|-------------|-------------------------------------------------|---------------------------------------------------------------------------------------|----------------------------------------------------------| +| ----------- | ----------------------------------------------- | ------------------------------------------------------------------------------------- | -------------------------------------------------------- | +| 2023/9/15 | [Informal Systems](https://informal.systems/) | [v1.0.0-rc6](https://github.com/celestiaorg/celestia-app/releases/tag/v1.0.0-rc6) | [informal-systems.pdf](docs/audit/informal-systems.pdf) | +| 2023/10/17 | [Binary Builders](https://binary.builders/) | [v1.0.0-rc10](https://github.com/celestiaorg/celestia-app/releases/tag/v1.0.0-rc10) | [binary-builders.pdf](docs/audit/binary-builders.pdf) | ## Careers diff --git a/docs/architecture/adr-013-non-interactive-default-rules-for-zero-padding.md b/docs/architecture/adr-013-non-interactive-default-rules-for-zero-padding.md index 29191b7a44..406a321c7c 100644 --- a/docs/architecture/adr-013-non-interactive-default-rules-for-zero-padding.md +++ b/docs/architecture/adr-013-non-interactive-default-rules-for-zero-padding.md @@ -49,12 +49,12 @@ To recap, the `SubtreeRootThreshold` determines the index of where a blob must s For example, assume `SubtreeRootThreshold = 64`. This would mean that the blobs smaller than the `64` can start at an index that is a multiple of one and therefore introduce zero padding. Blobs that are larger than `64` but smaller than `64 * 2 = 128` can use an index that is a multiple of 2 to get a maximum of 1 padding share. Blobs that are larger than `64 * 2 = 128` but smaller than `64 * 4 = 256` can use an index that is a multiple of 4 to get a maximum of 3 padding shares and so on. -Blob size (in number of shares) | Subtree root width[^2] | Index in square | Worst case padding ---------------------------------|------------------------|-----------------|------------------- -blob <= 64 | 1 | multiple of 1 | 0 -64 < blob <= 128 | 2 | multiple of 2 | 1 -128 < blob <= 256 | 4 | multiple of 4 | 3 -256 < blob <= 512 | 8 | multiple of 8 | 7 +| Blob size (in number of shares) | Subtree root width[^2] | Index in square | Worst case padding | +|---------------------------------|------------------------|-----------------|--------------------| +| blob <= 64 | 1 | multiple of 1 | 0 | +| 64 < blob <= 128 | 2 | multiple of 2 | 1 | +| 128 < blob <= 256 | 4 | multiple of 4 | 3 | +| 256 < blob <= 512 | 8 | multiple of 8 | 7 | Note: the threshold doesn't need to be `64` and the implementation versions this constant so that it is possible to modify over time. diff --git a/docs/architecture/adr-015-namespace-id-size.md b/docs/architecture/adr-015-namespace-id-size.md index d92a283449..3333c05f0d 100644 --- a/docs/architecture/adr-015-namespace-id-size.md +++ b/docs/architecture/adr-015-namespace-id-size.md @@ -61,21 +61,21 @@ The namespace ID must provide at least 72 bits of randomness [^2] to satisfy cri Another way to analyze this criteria is to determine the probability of duplicates if there exist N randomly generated namespaces. Columns in the table below represent the approximate probability that a collision would occur if N (e.g. 1 billion) random namespaces are generated.[^3] -Namespace ID size (bytes) | 1 billion (10^9) | 1 trillion (10^12) | 1 quadrillion (10^15) | 1 quintillion (10^18) ---------------------------|------------------|--------------------|-----------------------|---------------------- -8 | ~0.02674 | 1 | 1 | 1 -16 | 0 | ~1.4432e-15 | ~1.4693e-9 | ~0.00147 -20 | 0 | 0 | 0 | ~3.4205e-13 -32 | 0 | 0 | 0 | 0 +| Namespace ID size (bytes) | 1 billion (10^9) | 1 trillion (10^12) | 1 quadrillion (10^15) | 1 quintillion (10^18) | +|---------------------------|------------------|--------------------|-----------------------|-----------------------| +| 8 | ~0.02674 | 1 | 1 | 1 | +| 16 | 0 | ~1.4432e-15 | ~1.4693e-9 | ~0.00147 | +| 20 | 0 | 0 | 0 | ~3.4205e-13 | +| 32 | 0 | 0 | 0 | 0 | > As a rule of thumb, a hash function with range of size N can hash on the order of sqrt(N) values before running into collisions.[^4] -Namespace ID size (bytes) | Hash function range | Can hash this many items before running into collision ---------------------------|---------------------|------------------------------------------------------- -8 | 2^64 | 2^32 = ~4 billion items -16 | 2^128 | 2^64 = ~1 quintillion items -20 | 2^160 | 2^80 = ~1 septillion items -32 | 2^256 | 2^128 = ~3.4 quintillion items +| Namespace ID size (bytes) | Hash function range | Can hash this many items before running into collision | +|---------------------------|---------------------|--------------------------------------------------------| +| 8 | 2^64 | 2^32 = ~4 billion items | +| 16 | 2^128 | 2^64 = ~1 quintillion items | +| 20 | 2^160 | 2^80 = ~1 septillion items | +| 32 | 2^256 | 2^128 = ~3.4 quintillion items | ### Criteria 2 @@ -104,23 +104,23 @@ There are some tradeoffs to consider when choosing a namespace ID size. The namespace is prefixed to each NMT data leaf. Two namespaces are prefixed to each NMT non-leaf hash. Therefore, the nodes of an NMT will be larger based on the namespace size. Assuming shares are 512 bytes: -Namespace size (bytes) | NMT data leaf size (bytes) | NMT inner node size (bytes) ------------------------|----------------------------|---------------------------- -8 | 8 + 512 = 520 | 2*8 + 32 = 48 -16 | 16 + 512 = 528 | 2*16 + 32 = 64 -20 | 20 + 512 = 532 | 2*20 + 32 = 72 -32 | 32 + 512 = 544 | 2*32 + 32 = 96 +| Namespace size (bytes) | NMT data leaf size (bytes) | NMT inner node size (bytes) | +|------------------------|----------------------------|-----------------------------| +| 8 | 8 + 512 = 520 | 2*8 + 32 = 48 | +| 16 | 16 + 512 = 528 | 2*16 + 32 = 64 | +| 20 | 20 + 512 = 532 | 2*20 + 32 = 72 | +| 32 | 32 + 512 = 544 | 2*32 + 32 = 96 | ### NMT proof size Increasing the size of NMT nodes will increase the size of the NMT proof. Assuming shares are 512 bytes, square size is 128, the NMT for a row will contain 2 * 128 leaves. If the NMT proof is for a single leaf: -Namespace size (bytes) | Unencoded NMT proof size (bytes) | Protobuf encoded NMT proof size (bytes) | Protobuf encoded NMT proof with [gzip](https://pkg.go.dev/compress/gzip) (bytes) ------------------------|----------------------------------|-----------------------------------------|--------------------------------------------------------------------------------- -8 | 336 | 354 | 382 -16 | 448 | 466 | 408 -20 | 504 | 522 | 466 -32 | 672 | 690 | 630 +| Namespace size (bytes) | Unencoded NMT proof size (bytes) | Protobuf encoded NMT proof size (bytes) | Protobuf encoded NMT proof with [gzip](https://pkg.go.dev/compress/gzip) (bytes) | +|------------------------|----------------------------------|-----------------------------------------|----------------------------------------------------------------------------------| +| 8 | 336 | 354 | 382 | +| 16 | 448 | 466 | 408 | +| 20 | 504 | 522 | 466 | +| 32 | 672 | 690 | 630 | Note: if the NMT proof is an absence proof, an additional leaf node is included in the proof. @@ -147,12 +147,12 @@ If the namespace size is increased, the maximum possible blob will decrease. Giv If the namespace size is increased, whenever a SHA256 invocation takes place (e.g. NMT's [HashNode](https://github.com/celestiaorg/nmt/blob/fd00c52175c48bad64d03444689162fb9c6bee41/hasher.go#L265)), more data needs to be SHA256'ed. For example: -Namespace size (bytes) | [HashNode](https://github.com/celestiaorg/nmt/blob/fd00c52175c48bad64d03444689162fb9c6bee41/hasher.go#L302)'s SHA256 max data (bytes) ------------------------|-------------------------------------------------------------------------------------------------------------------------------------- -8 | 97 -16 | 129 -32 | 193 -33 | 197 +| Namespace size (bytes) | [HashNode](https://github.com/celestiaorg/nmt/blob/fd00c52175c48bad64d03444689162fb9c6bee41/hasher.go#L302)'s SHA256 max data (bytes) | +|------------------------|---------------------------------------------------------------------------------------------------------------------------------------| +| 8 | 97 | +| 16 | 129 | +| 32 | 193 | +| 33 | 197 | The data size is calculated as follows: `SHA256(domain_sep || left_min || left_max || left_data_hash || right_min || right_max || right_data_hash)` = 1 + namespaceSize + namespaceSize + 32 + namespaceSize + namespaceSize + 32 which simplifies to (4 \* namespaceSIze) + (2 \* 32) + 1. @@ -161,12 +161,12 @@ Hashing has a high performance impact in ZK contexts. Since a larger namespace s Based on this StackOverflow [post](https://crypto.stackexchange.com/questions/54852/what-happens-if-a-sha-256-input-is-too-long-longer-than-512-bits), when the data provided to SHA256 exceeds 56 bytes, the data must be chunked into [64 byte blocks](https://cs.opensource.google/go/go/+/refs/tags/go1.20.4:src/crypto/sha256/sha256.go;l=29;drc=995c0f310c087c9cbc49112ecc48459a96310451) with a trailing 56 byte block + 8 bytes to store the original data length as a `uint64`. -Namespace size (bytes) | [HashNode](https://github.com/celestiaorg/nmt/blob/fd00c52175c48bad64d03444689162fb9c6bee41/hasher.go#L302)'s SHA256 max data (bytes) | SHA256 compression invocations ------------------------|---------------------------------------------------------------------------------------------------------------------------------------|------------------------------- -N/A | 56 | 1 -0 to 13 | 64 + 56 = 120 | 2 -14 to 29 | 64 + 64 + 56 = 184 | 3 -30 to 45 | 64 + 64 + 64 + 56 = 248 | 4 +| Namespace size (bytes) | [HashNode](https://github.com/celestiaorg/nmt/blob/fd00c52175c48bad64d03444689162fb9c6bee41/hasher.go#L302)'s SHA256 max data (bytes) | SHA256 compression invocations | +|------------------------|---------------------------------------------------------------------------------------------------------------------------------------|--------------------------------| +| N/A | 56 | 1 | +| 0 to 13 | 64 + 56 = 120 | 2 | +| 14 to 29 | 64 + 64 + 56 = 184 | 3 | +| 30 to 45 | 64 + 64 + 64 + 56 = 248 | 4 | Note: to verify the number of SHA256 compression invocations, we analyzed the number of loop executions inside the Golang SHA256 implementation [here](https://github.com/golang/go/blob/96add980ad27faed627f26ef1ab09e8fe45d6bd1/src/crypto/sha256/sha256block.go#L83) and it matches the expected number of invocations in the table above. See raw data [here](https://gist.github.com/rootulp/4cfc10c1c80a15cc57f0b35f330ac542). diff --git a/docs/architecture/adr-019-strict-inflation-schedule.md b/docs/architecture/adr-019-strict-inflation-schedule.md index 8102ad56aa..3805de02d3 100644 --- a/docs/architecture/adr-019-strict-inflation-schedule.md +++ b/docs/architecture/adr-019-strict-inflation-schedule.md @@ -119,8 +119,8 @@ Q: What is the skew between `BlocksPerYear` on popular Cosmos SDK chains and the Cosmos Hub (a.k.a Gaia) sets `BlocksPerYear` to 4,360,000 [here](https://github.com/cosmos/gaia/blob/8a522e98a2863205cf02fb97f8ad27d091670b9d/docs/governance/current-parameters.json#L86). Cosmos Hub has a block time 6.353 ms. Numia data doesn't have complete data for 2021 so we'll examine 2022: -Year | BlocksPerYear | Actual # of Blocks | Skew ------|---------------|--------------------|------ -2022 | 4,360,000 | 4,580,463 | 5.05% +| Year | BlocksPerYear | Actual # of Blocks | Skew | +|------|---------------|--------------------|-------| +| 2022 | 4,360,000 | 4,580,463 | 5.05% | Source: [Numia query](https://console.cloud.google.com/bigquery?sq=611612269782:f0c42f9584c448c78a4ec5f118c2091c)