v0.4.0

v0.4.0 | 2022-10-17

This release includes large refactorings, breaking changes and improvements related to four primary work-streams: building out the public API, hardening HeaderExchange, refactoring blocksync architecture and increasing metrics coverage.

Most notably, this release contains:

• 2 breaking changes, in addition to other improvements, to HeaderExchange as part of our work to harden and optimise it
• A large-scale refactoring to the structure of the new nodebuilder package (formerly node) as well as the way components are constructed/managed, in an effort to clarify the boundaries between components and create a cleaner foundation for building out the node's API
• Upgrade to the latest breakingcelestia-app dependency, including using instance of app instead of kvstore in our integration tests (located in nodebuilder/tests)
• Metrics coverage for das and fraud packages, in addition to increased coverage for other key celestia-node behaviours
• Consolidation of ipld and share packages into a cleaner, better defined structure, as well as general improvements to ipld to remove redundant/unnecessary behaviours

What's Changed

BREAKING

• feat!(header/p2p): add deadlines on stream and rework proto by @vgonkivs in #1038
• feat!(header/p2p): extend header response with status code by @vgonkivs in #1135
• chore!: upgrade to celestia-app v0.7.0 by @rootulp in #1147

FEATURES

• feat(metrics): metrics for PFD transactions, sampling, and broadcasts by @distractedm1nd in #1083
• feat(das): add metrics to DASer by @walldiss in #1125
• feat(fraud): add metrics to fraud package by @vgonkivs in #1047
• feat(nodebuilder): add AccountAddress to state's Module by @Bidon15 in #1209
• feat(header/p2p): implement ErrHeadersLimitExceeded error by @vgonkivs in #1153

IMPROVEMENTS

• refactor(node/state): Refactor node pkg -> nodebuilder and split services into modules by @distractedm1nd @renaynay in #997
• refactor(nodebuilder): Move parse methods and flag definitions to nodebuilder subpackages by @distractedm1nd in #1161
• refactor(nodebuilder): moving service/ services to respective node sub-packages by @distractedm1nd in #1056
• fix(state): returning CoreAccessor from nodebuilder for state.WithMetrics by @distractedm1nd in #1200
• refactor(header/p2p): request head from multiple peers by @vgonkivs in #1046
• refactor(swamp): use celestia-app instead of the kvstore by @evan-forbes in #1160
• refactor(share): Consolidate share pkg by @walldiss in #1177
• refactor(share): IPLD/NMT improvements by @walldiss in #1223

BUG FIXES

• bugfix(fraud): close stream properly by @vgonkivs in #1152
• bugfix(nodebuilder/fraud): remove fraud.Subscriber from module by @vgonkivs in #1169
• bugfix: fix flakiness in TestFullReconstructFromLights by @vgonkivs in #1171
• fix(ipld/plugin): don't truncate a type byte when it's not in the data by @Wondertan in #1196
• fix(nodebuider/header): initialize Store during start lifecycle instead of fx.Invoke by @Wondertan in #1218
• fix(nodebuilder): fix broken shutdown for services on fraud proofs receival by @distractedm1nd in #1220

MISC

• log(header): Add detailed error for non-adjacent header attempt by @renaynay in #1123
• log(das): add extra debug logs for sampled headers by @walldiss in #1157
• chore(deps): bump go.opentelemetry.io/otel/metric from 0.32.0 to 0.32.1 by @dependabot in #1167
• chore(deps): bump codecov/codecov-action from 3.1.0 to 3.1.1 by @dependabot in #1164
• chore(deps): bump go.uber.org/fx from 1.18.1 to 1.18.2 by @dependabot in #1179
• Disable verbose log on Debug Level for badger, watchdog, and basichost. by @HoytRen in #1180
• Fix markdown lint workflow to run on directories by @adlerjohn in #1170
• fix (.github/workflows): Skip over integration tests in unit test action by @renaynay in #1193
• fix: README link for header doc by @rootulp in #1192
• nodebuilder: Remove current borked RPC tests in favour of implementing better ones with #962 by @renaynay in #1190
• docs/adr: add daser parallelization adr by @walldiss in #1096
• chore: disable debug log in Makefile by @Wondertan in #1210
• fix(nodebuilder/header): Only provide p2p.Subscriber as p2p.Broadcaster for bridge node by @renaynay in #1203
• chore(nodebuilder/header): remove dead metrics code by @Wondertan in #1217
• Remove watchdog since go 1.19 don't need it anymore. by @HoytRen in #1212
• chore(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.0 by @dependabot in #1222
• .github: add new ci_release workflow to unify CI by @MSevey in #1191

New Contributors

• @HoytRen made their first contribution in #1180
• @MSevey made their first contribution in #1191

Full Changelog: v0.3.1...v0.4.0

v0.3.1

v0.3.1 - DASer parallelisation and quick fix to cel-key

This patch release notably includes an optimisation to the DASer in order to provide faster sampling over past headers and a fix to the cel-key utility to read from the correct node directory (which was broken in v0.3.0).

What's Changed

FEATURES

• feat(rpc): implementing staking reads and replace sdk.Address by @distractedm1nd in #1060

IMPROVEMENTS

• refactor(service/state): CoreAccessor relies on header.Head instead of header.Getter by @renaynay in #1088
• das: Parallelise DASer by @walldiss in #988
• header/store: use an atomic.Pointer instead of Lock by @Martichou in #1079

BUG FIXES

• fix(cel-key): reading from correct directory by @distractedm1nd in #1087
• refactor(service/state): Fix log to return correct height of balance request by @renaynay in #1089

DEPENDENCIES

• chore(deps): bump github.com/multiformats/go-multiaddr from 0.6.0 to 0.7.0 by @dependabot in #1071
• chore(deps): bump go.opentelemetry.io/otel/sdk from 1.9.0 to 1.10.0 by @dependabot in #1094
• chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.9.0 to 1.10.0 by @dependabot in #1098
• chore(deps): Upgrading to go 1.19 by @distractedm1nd in #1067
• chore(deps): bump go.opentelemetry.io/otel/metric from 0.31.0 to 0.32.0 by @dependabot in #1136
• refactor(service/state): Use DefaultProofRuntime instead of ibc-go VerifyMembership endpoint by @renaynay in #1131
• chore: remove deprecated linters by @rootulp in #1150
• chore(deps): bump github.com/ipfs/go-cid from 0.3.0 to 0.3.2 by @dependabot in #1053

MISC

• typo: Fix the cmd grpc port in the docs by @sweexordious in #1097
• README: ✍️ added Table of Contents by @nitinmewar in #1082
• Remove Changelog section from CONTRIBUTING.md by @rootulp in #1148
• docs(adr): BlockSync Overhaul ADR #11 - Part 1 by @Wondertan in #1037
• Remove @liamsi from global CODEOWNERS by @Wondertan in #1095
• header/p2p: remove ExtendedHeaderRequest struct by @vgonkivs in #1141
• Add Protobuf section to CONTRIBUTING.md by @rootulp in #1145

New Contributors

• @sweexordious made their first contribution in #1097
• @walldiss made their first contribution in #988
• @Martichou made their first contribution in #1079

Full Changelog: v0.3.0...v0.3.1

v0.3.0

v0.3.0 | 2022-09-06

Arabica Testnet Release

It has been half a year since our last minor release, and, as a result, our third minor release of celestia-node introduces a large range of new and hardened functionality. Some key highlights can be found below:

Highlights

• Re-introduction of full node type
  • Operates solely on the p2p network and independently from the Core network
  • Able to sync and reconstruct block data from light nodes only
  • Secures light node by generating and broadcasting fraud proofs
  • Light nodes discover Full Nodes over p2p network via DHT
• Implementation of state interaction via a celestia-core connection (ability to read and verify balances, as well as submit transactions to the celestia network), specifically the ability to submit a PayForData transaction
• Data availability sampling (DASing) over headers in the past and local caching of successful sampling results
• Introduction of FraudService
• • Bad Encoding Fraud Proofs (BEFPs)
• Alpha node API
• Key management utility via cel-key
• Introduction of telemetry via opentelemetry
• Parallelized GetSharesByNamespace resulting in faster application data retrieval
• Removes Core Node embedding for Bridge Node

Migration

The Node's Store default directory path has changed, and If you have been running any of the past two release candidates against mamaki and want to retain your data, please run the Node with --node.store ~/.celestia-<node_type> flag or migrate your store from $HOME/.celestia-<node_type> to $HOME/.celestia-<node_type>-mamaki.

BREAKING CHANGES

• feat!(header/p2p/exchange): Upgrade and reformat header exchange protocol ID
• refactor!(cmd/cel-key): Extract cel-key utility into separate binary
• refactor!(cmd|node): Refactor core flags

FEATURES

• feat(node): Introduce full node type
• feat(service/state): Introduce StateService and CoreAccessor
• feat(das): Implement DASing of past headers
• feat(fraud): Implement Bad Encoding Fraud Proofs (BEFPs)
• feat(ipld|shares): Rework data retrieval
• feat(service/state): Add /submit_pfd endpoint
• feat(service/rpc): Add rpc service to gather all available endpoints | alpha API
• feat(node/state): Automatic key generation if no key found in keystore
• feat(service/share): Implement CacheAvailability
• feat(service/share): Implement full node discovery
• feat(fraud): Implement fraud service
• feat(node): Integrate fraud service
• feat(fraud): Implement fraud store
• feat(das): Implement DASState
• feat(cmd): Integration of OpenTelemetry with OTLP exporter
• feat(service/state): Implement balance verification against AppHash
• feat(service/state): Transaction proxy for staking module tx types
• feat(fraud): implement fraud sync

IMPROVEMENTS

• refactor(service/share): No error if namespace not found in GetSharesByNamespace
• refactor(header): Move header sub-services to different directories
• test(service/share): Add full node reconstruction tests
• dep(ipld): Move from DAGService to BlockService
• refactor(ipld): Rework retriever
• refactor(fraud): Extend proof structure
• feat(service/share): Handle disconnection of discovered peer
• feat(ipld): Concurrent GetSharesByNamespace
• refactor(cmd): Use context.WithValue instead of custom Env implementation
• feat(cmd): Provide ability to specify network to user
• feat|refactor(header/sync): network head determination
• params: Switch default network to arabica

BUG FIXES

• fix(service/header): Unlock heightSub lock on rare cases
• fix(core): Prevent spamming of old headers during bridge node sync
• fix(service/share): Fix GetSharesByNamespace ordering
• fix(ipld): Concurrent GetShares to prevent blocking of share fetching in case of one unavailable share
• fix(ipld): Fix race in rare case when Retriever fetches from both row and col
• fix(ipld): Rely on rsmt2d.Codec constructor to avoid races
• fix(node): Invoke function to ensure empty block data is stored at least once
• fix(das): Return from sampling routine upon failure
• fix(das): Fix checkpoint in catchUp manager
• fix(service/header): Fix header store design issues with Batch
• fix(ipld): Re-enable re-importing of rsmt2d.ExtendedDataSquare
• fix(ipld): Fix races in doRequest method
• fix(service/share): Protect datastore autobatch with a lock
• fix(node/p2p): Add condition to restore routing table refresh period
• fix(node): Change applied Availability for bridge node
• fix(fraud): Blocklist peer if sent invalid fraud proof

MISCELLANEOUS

• refactor(core): Remove embedded core node option
• chore(ipld|shares): cleanup Share types, global default rsmt2d codec, various cleanups
• refactor(header): Exchange interface implements Getter
• feat(service/share): Make discovery constants configurable

New Contributors

• @rootulp made their first contribution in #479
• @toanngosy made their first contribution in #496
• @adlerjohn made their first contribution in #605
• @liamsi made their first contribution in #607
• @0xyicheng made their first contribution in #767
• @distractedm1nd made their first contribution in #852
• @iofq made their first contribution in #769

Full Changelog: https://github.com/celestiaorg/celestia-node/compare/v0.2.0...v0.3.0## v0.3.0 | 2022-09-06

v0.3.0-rc2

Another release candidate for the almost complete Full Node feature.

v0.3.0-rc1

This is a pre-release for the Mamaki testnet. The full changelog will be available on the release.

v0.2.0

v0.2.0 | 2022-02-23

The second minor release focuses on the stability and robustness of existing features via refactorings, mainly for
headers synchronization and storing, while fixing the DA network segregation by providing bootstrap peers, allowing you to
run a Light Node with zero effort. The release also includes a few breaking changes.

Highlights

Hardcoded bootstrap peers

Bootstrappers are necessary for peers to connect to when they come online in order to be part of the DA p2p network.
Additionally, they serve block headers and shares for everyone, so any other node joining the network can use them to
synchronize headers and perform Data Availability Sampling. Now it's unnecessary to maintain your own Bridge(previously
Full) Node to run a Light Node, as now it will rely on bootstrappers by default.

Zero effort Light Node

Now becoming a valuable node that contributes to the security of the of Celestia network is only two commands away. Simply run celestia light init followed by celestia light start, and you are good to go. Still, you can keep running your own Bridge Node as the trusted peer.

Test Swamp

The release comes with a new internal testing library for Celestia Node called Test Swamp, aimed to provide an
ergonomic framework for integration tests for celestia-node. It allows simulating a network on which we can test various
high-level scenarios.

Header Synchronization

It was almost rewritten from scratch with a better design which:

• Tolerates network disconnections
• Tolerates primary network interface changes
  • both above are useful for Light Node running on laptops or mobile phones
• Optimizes bandwidth and IO usage, subsequently speeding up the synchronization time for ~20%
• Eliminates long-standing issue with header duplicates flooding the network
• Fixes the most common issue community has faced - constant logging of invalid headers error.

Renamed Full Node

Full Node is now called Bridge Node to emphasize its purpose of bridging the core consensus and the halo DA networks, both
powering the Celestia project.

Spoiler: Next release will come with a reincarnation of the Full Node type, which operates only over the DA celestia-node network

Full node operators after the update should now migrate to Bridge Node. This migration is trivial:

• Rename .celestia-full to .celestia-bridge
• Change scripts from celestia full to celestia bridge

Trusted Peers

• trusted-peer is now trusted-peers as the flag now allows passing multiple trusted peers.
• Config.Services.TrustedPeer is now Config.Services.TrustedPeers

BREAKING CHANGES

• node: Light node can be initialised with multiple trusted peers #455 @vgonkivs
• chore: rename Repository to Store #296 @Wondertan
• chore: rename Full node to Bridge node #294 @Wondertan
• node: remove InitWith #291 @Wondertan

FEATURES

• feat(cmd/cel-shed): new header category and store-init cmd #462 @Wondertan
• feat(cmd): cli flag to enable http/pprof handler to capture profiles #463 @Wondertan
• service/header: SyncState #397 @Wondertan
• feat(service/header): HeightSub #428 @Wondertan
• params: Define Network Types #346 @Wondertan
• feat(cmd): give a birth to cel-shed and p2p key utilities #281 @Wondertan
• feat(cmd|node): MutualPeers Node option and CLI flag #280 @Wondertan
• node: enhance DI allowing overriding of dependencies @Wondertan
• ci: create docker build GH action @jbowen93
• swamp: initial structure of the tool @Bidon15

IMPROVEMENTS

• feat(node): add go-watchdog to curb OOMs #466 @Wondertan
• perf(node/store): fine-tune Badgerdb params #465 @Wondertan
• feat(service/header): update Store.Append to return amount of applied/valid headers #434 @Wondertan
• refactor(service/header): rework on disk writing strategy of the Store #431 @Wondertan
• refactor(service/header): extract store initialization from Syncer #430 @Wondertan
• header: hardening syncing logic #334 @Wondertan
• feat(params): add bootstrappers #399 @Wondertan
• service/header: remove start/stop from P2PExchange @Bidon15
• service/share: Implement FullAvailability @renaynay
• services/header: Refactor HeaderService to be responsible for broadcasting new ExtendedHeaders to the gossipsub network @renaynay
• cmd: introduce Env - an Environment for CLI commands #313 @Wondertan
• node: Adding WithHost options to settings section #301 @Bidon15
• node: Adding WithCoreClient option #305 @Bidon15
• service/header: Refactor HeaderService to only manage its sub-services' lifecycles #317 @renaynay
• docker: Created docker/ dir with Dockerfile and entrypoint.sh script @jbowen93
• chore(share): handle rows concurrently in GetSharesByNamespace #241 @vgonkivs
• ci: adding data race detector action @Bidon15
• node: add the cmdnode.HeadersFlags() to the Bridge Node's init and start commands #390 @jbowen93

BUG FIXES

• fix(service/header): lazily load Store head #458 @Wondertan
• fix(service/header): allow some clock drift during verification #435 @Wondertan
• service/header: fix ExtendedHeader message duplicates on the network #409 @Wondertan
• fix(header/service): #339 race @Wondertan
• core: Properly fetch Validators from Core and two more fixes #328 @Wondertan
• header: Added missing err value in ErrorW logging calls @jbowen93
• service/block, node/p2p: Fix race conditions in TestExtendedHeaderBroadcast and TestFull_P2P_Streams. @jenyasd209
• ci: increase tokens ratio for dupl to fix false positive scenarios @Bidon15
• node: Wrap datastore with mutex to prevent data race @Bidon15
• ci: update Docker entrypoint.sh to use new store.path flag name @jbowen93
• docker: update docker/entrypoint.sh to use new node.store flag replacing store.path #390 @jbowen93

MISCELLANEOUS

• chore: bump deps #297 @Wondertan
• workflows/lint: update golangci-lint to v1.43 #308 @Wondertan
• [feat(node): extrac...

v0.2.0-rc1

This is a release candidate for the upcoming v0.2.0 release. Detailed info about the release will come together with it.

v0.1.1

v0.1.1 | 2022-01-07

A quick hot-fix release to enable Full Node sync reliably with validator set bigger than 30.

Bug Fixes

• Properly fetch Validators from Core and two more fixes #328 @Wondertan

v0.1.0

v0.1.0 | 2021-11-03

This is the first celestia-node release. The release constitutes the foundation for the data availability "halo" network that will complement the core consensus network.

Mainly, we introduce 3 packages:

• service/header - defines everything related to syncing, exchanging, and storing chain headers.
• service/share - defines everything related to sampling and requesting Shares (chunks of erasure-coded block data) from the network
• node - defines the Node singleton with its assembly, essentially gluing all the bits together.

For further information regarding the architecture and features introduced in this release, refer to the devnet ADR.

FEATURES

• das: Log out square width after sampling #254 @Wondertan
• das: log time it takes to complete as sampling routine for a header #238 @renaynay
• node/p2p: Deduplicate PubSub msgs #239 @Wondertan
• service/header: more descriptive err log for append in store.go #231 @renaynay
• cmd, service/header: rename genesis to trustedHash to avoid confusion, extract flags to common var for re-use #229 @renaynay
• service/header: Verify hash matches header retrieved in RequestByHash() #224 @renaynay
• Genesis: The PR to make testnet happen! #221 @Wondertan
• feat(cmd): allow setting log level for start comamnd #217 @Wondertan
• service/header: Header Exchange implemented over Core Node #216 @renaynay
• Persist generated key to keystore and reuse existing key if present #213 @jbowen93
• service/header: implement basic syncer #212 @Wondertan
• node: add p2p multiaddr log #204 @bidon15
• service/block, service/header: BlockService broadcasts generated ExtendedHeaders to network #203 @renaynay
• cmd: add flag to start node on remote core #196 @vgonkivs
• service/header: Store implementation #195 @Wondertan
• service/header: Implement HeaderExchange #188 @renaynay
• das: implement DASer #177 @Wondertan
• share: Availability interface and implementation #171 @Wondertan
• service/share, ipld: adds ability to retrieve shares by namespace ID #170 @renaynay
• service/share: Service interface and basic implementation for GetShare #167 @Wondertan
• service/block: add Commit from current block in ExtendedHeader #165 @renaynay
• service/share, ipld: adds ability to retrieve shares by namespace ID #160 @renaynay
• Add version command #159 @orlandorode97
• node: start and stop block service by notification from lifecycle#158 @vgonkivs
• service/header: extension of ExtendedHeader and protobuf serialization #153 @Wondertan
• service/header: Implement basic header Service structure and interfaces #148 @renaynay
• node: add cache for blockstore #136 @vgonkivs
• MAKEFILE: make build #133 @renaynay
• ipld: adding IPLD package to provide helpers for dealing with IPLD-based data storage #110 @renaynay
• feat(node): unified node constructor #104 @Wondertan
• cmd: main function, full and light commands with subcommands #97 @Wondertan
• service/block: add basic encoding validation in BlockService #96 @renaynay
• node: provide DAGService #95 @Wondertan
• node: use Repository on the Node and slightly improve testing utilites #94 @Wondertan
• service/header, service/block: Implements spec'd out types for block and header #93 @renaynay
• node: initialization and Repository #86 @Wondertan
• core: initialization and Repository #81 @Wondertan
• service/block: new block listener + erasure coding #77 @renaynay
• node: updates for node.Config #74 @Wondertan
• libs/fslock: a simple utility to lock directories #71 @Wondertan
• libs/keystore: introduce Keystore a crypto key manager #66 @Wondertan
• block: create Fetcher interface #59 @renaynay
• node: add more p2p services and protocols #57 @Wondertan
• core: package managing Core Node #51 @Wondertan
• rpc: implement simple RPC client to dial Celestia Core endpoints #48 @renaynay
• node: Implement basic Node #29 @Wondertan