Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: pull cometbft v0.34.35 #1495

Merged
merged 179 commits into from
Sep 30, 2024
Merged

Conversation

rootulp
Copy link
Collaborator

@rootulp rootulp commented Sep 20, 2024

Closes #1468 by pulling upstream v0.34.35 with these notable merge conflicts:

  • Our repo upgraded Go past what upstream has so I choose to retain the more recent Go version

And these changes to make unit tests pass:

  1. Refactor CI to run all tests in one execution. It takes the same time (4 mins) as it previously did. But now it's easier to reason about the flakes.
  2. Included refactor: TestTxPool_ConcurrentTxs #1503
  3. Included fix: TestTxPool_ConcurrentlyAddingTx #1501

FLUPs

  1. Make it easier to pull upstream changes #1504
  2. Opens TestTxMempool_ExpiredTxs_Timestamp flake #1502 and skips that test

Testing

I used go mod replace and verified that celestia-app can use this version of celestia-core.

  • single-node.sh
  • mocha.sh

mergify bot and others added 30 commits May 1, 2023 21:08
…#766)

* Fix `TestStateOversizedBlock` (celestiaorg#755)

* Fix TestStateOversizedBlock

* Moved `findBlockSizeLimit` together with other aux functions

(cherry picked from commit c58597d656d5c816334aff9ea8e600bdbc534817)

# Conflicts:
#	consensus/state_test.go

* Revert "Fix `TestStateOversizedBlock` (celestiaorg#755)"

This reverts commit e7d2a127a26d4d833bae3aef0917bb6bca2cda5b.

* Fix `TestStateOversizedBlock` (celestiaorg#755)

* Fix TestStateOversizedBlock

* Moved `findBlockSizeLimit` together with other aux functions

---------

Co-authored-by: Sergio Mena <[email protected]>
…#788)

(cherry picked from commit ecd5ee112533cda28900cbd75afb349f67da3fa5)

Co-authored-by: Sergio Mena <[email protected]>
…aorg#794)

* Unsafe int cast in `kill` command (celestiaorg#783)

* Unsafe int cast in `kill` command

* Revert "Unsafe int cast in `kill` command"

This reverts commit bbd649bd372ca90f83dea7b424d67dafbd9eb541.

* Changed strategy

(cherry picked from commit 03c5e7727a03983b54623e731d5d3d8dd4ac75ec)

# Conflicts:
#	cmd/cometbft/commands/debug/kill.go

* Revert "Unsafe int cast in `kill` command (celestiaorg#783)"

This reverts commit b7ab279a6df1f062bec60bcf95947d2a87f4ccec.

* Unsafe int cast in `kill` command (celestiaorg#783)

* Unsafe int cast in `kill` command

* Revert "Unsafe int cast in `kill` command"

This reverts commit bbd649bd372ca90f83dea7b424d67dafbd9eb541.

* Changed strategy

---------

Co-authored-by: Sergio Mena <[email protected]>
…lestiaorg#805)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.17.0 to 1.18.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.17.0...v1.18.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#798)

Replaced int64 with big.int

Co-authored-by: Lasaro <[email protected]>
Co-authored-by: Sergio Mena <[email protected]>
Co-authored-by: Thane Thomson <[email protected]>
…elestiaorg#870)

Bumps [slackapi/slack-github-action](https://github.com/slackapi/slack-github-action) from 1.23.0 to 1.24.0.
- [Release notes](https://github.com/slackapi/slack-github-action/releases)
- [Commits](slackapi/slack-github-action@v1.23.0...v1.24.0)

---
updated-dependencies:
- dependency-name: slackapi/slack-github-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lestiaorg#871)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.18.0 to 1.19.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.18.0...v1.19.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lestiaorg#909)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.19.0 to 1.20.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.19.0...v1.20.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
celestiaorg#890) (celestiaorg#927)

* add a test to trigger the issue
* add a fix (in particular, we track the sender when receiving a tx twice)
* add a changelog
* update fix and test wrt. v0.34.x

---------
Co-authored-by: Daniel <[email protected]>
Co-authored-by: Pierre Sutra <[email protected]>
…ackport celestiaorg#865) (celestiaorg#970)

* fix: avoid recursive call after rename to (*PeerState).MarshalJSON (celestiaorg#865)

* avoid recursive call after rename to (*PeerState).MarshalJSON

* add test

* add change doc

* explain for nolint

* fix lint

* fix golangci-lint to v1.52.2

* fix golangci-lint to v1.52.2

* Revert "fix golangci-lint to v1.52.2"

This reverts commit 598a9ef4c86fc29cf038251676c33a222217826c.

* Revert "fix golangci-lint to v1.52.2"

This reverts commit a8aad121e27382813e95b1911b1b560c62e1c7c3.

* Reintroduced `cmtjson`

* Avoid copying Mutex

* Avoid copying Mutex -- 2nd try, more succint

* Update .changelog/unreleased/bug-fixes/865-fix-peerstate-marshaljson.md

* Update consensus/reactor_test.go

---------

Co-authored-by: Sergio Mena <[email protected]>
(cherry picked from commit f6ea09171a2bf9f695f59b65f5c51e4a8c168015)

# Conflicts:
#	consensus/reactor_test.go

* Revert "fix: avoid recursive call after rename to (*PeerState).MarshalJSON (celestiaorg#865)"

* fix: avoid recursive call after rename to (*PeerState).MarshalJSON (celestiaorg#865)

* avoid recursive call after rename to (*PeerState).MarshalJSON

* add test

* add change doc

* explain for nolint

* fix lint

* fix golangci-lint to v1.52.2

* fix golangci-lint to v1.52.2

* Revert "fix golangci-lint to v1.52.2"

This reverts commit 598a9ef4c86fc29cf038251676c33a222217826c.

* Revert "fix golangci-lint to v1.52.2"

This reverts commit a8aad121e27382813e95b1911b1b560c62e1c7c3.

* Reintroduced `cmtjson`

* Avoid copying Mutex

* Avoid copying Mutex -- 2nd try, more succint

* Update .changelog/unreleased/bug-fixes/865-fix-peerstate-marshaljson.md

* Update consensus/reactor_test.go

---------

Co-authored-by: Sergio Mena <[email protected]>

---------

Co-authored-by: mmsqe <[email protected]>
Co-authored-by: Sergio Mena <[email protected]>
* changelog: Clean up and reorder

Signed-off-by: Thane Thomson <[email protected]>

* changelog: Add missing entries

Signed-off-by: Thane Thomson <[email protected]>

* changelog: Release v0.34.29

Signed-off-by: Thane Thomson <[email protected]>

* Rebuild changelog

Signed-off-by: Thane Thomson <[email protected]>

* version: Bump to v0.34.29

Signed-off-by: Thane Thomson <[email protected]>

* test/e2e: Use Debian Bullseye as base image

Golang recently started offering Debian Bookworm as the default distro
for `golang:1.19`, which provides a newer version of RocksDB than what
we support in cometbft-db.

For now this pins the image to Bullseye, which is the base image we have
been using for some time now.

Signed-off-by: Thane Thomson <[email protected]>

---------

Signed-off-by: Thane Thomson <[email protected]>
…lestiaorg#940)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.20.0 to 1.21.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.20.0...v1.21.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…g#942)

Bumps [docker/login-action](https://github.com/docker/login-action) from 2.1.0 to 2.2.0.
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](docker/login-action@v2.1.0...v2.2.0)

---
updated-dependencies:
- dependency-name: docker/login-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…g#1000)

* Add `CMT_HOME` (or remove it?) (celestiaorg#983)

Closes celestiaorg#982

Added `CMT_HOME` everywhere `CMTHOME` is used.

### Notes to reviewers

This could be fixed the opposite way, by removing the only reference to `CMT_HOME` in the code, and also the reference in `UPGRADING.md` (two lines of code).

However, the reference in `UPGRADING.md`, which is part of our documentation, is already present in `v0.34.x` (not in `v0.37.x` though!). That's why this PR introduces `CMT_HOME` to work in equal conditions as `CMTHOME`.

If reviewers lean toward removing `CMT_HOME` from the doc in `v0.34.x` (and unreleased `v0.38.x` and `main`), I can do it easily.

---

#### PR checklist

- [x] Tests written/updated
- [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments

(cherry picked from commit b7be568f4a59edf41942ec7e0e25edd72df604d8)

# Conflicts:
#	cmd/cometbft/commands/root_test.go

* Revert "Add `CMT_HOME` (or remove it?) (celestiaorg#983)"

* Add `CMT_HOME` (or remove it?) (celestiaorg#983)

Closes celestiaorg#982

Added `CMT_HOME` everywhere `CMTHOME` is used.

This could be fixed the opposite way, by removing the only reference to `CMT_HOME` in the code, and also the reference in `UPGRADING.md` (two lines of code).

However, the reference in `UPGRADING.md`, which is part of our documentation, is already present in `v0.34.x` (not in `v0.37.x` though!). That's why this PR introduces `CMT_HOME` to work in equal conditions as `CMTHOME`.

If reviewers lean toward removing `CMT_HOME` from the doc in `v0.34.x` (and unreleased `v0.38.x` and `main`), I can do it easily.

---

- [x] Tests written/updated
- [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments

---------

Co-authored-by: Sergio Mena <[email protected]>
…rg#1012)

it'll make the handshake work with graceful shutdown(see: cosmos/cosmos-sdk#16202)

handshake could be a long running process if there are many local blocks to replay, for example we use it to do profiling.

Hope we can backport this to 0.34.x.

---

- [ ] Tests written/updated
- [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments
…lestiaorg#1025)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.21.0 to 1.22.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.21.0...v1.22.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
celestiaorg#1047)

Closes celestiaorg#666

This PR adds double quotes to `path` param of `/abci_query` endpoint.

---

#### PR checklist

- [ ] Tests written/updated
- [ ] Changelog entry added in `.changelog` (we use [unclog](https://github.com/informalsystems/unclog) to manage our changelog)
- [ ] Updated relevant documentation (`docs/` or `spec/`) and code comments

(cherry picked from commit f6f13b1f67a54549d9f212a859ca4924d6ad9127)

Co-authored-by: Steven Ferrer <[email protected]>
…lestiaorg#1070)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.22.0 to 1.23.1.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.22.0...v1.23.1)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…estiaorg#1069)

Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.7.0 to 2.8.0.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](docker/setup-buildx-action@v2.7.0...v2.8.0)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…estiaorg#1099)

Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.8.0 to 2.9.0.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](docker/setup-buildx-action@v2.8.0...v2.9.0)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…estiaorg#1127)

Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 2.9.0 to 2.9.1.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](docker/setup-buildx-action@v2.9.0...v2.9.1)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lestiaorg#1128)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.23.1 to 1.24.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.23.1...v1.24.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…lestiaorg#1162)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.24.0 to 1.25.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.24.0...v1.25.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* CV OnStop close evidenceStore

* CV OnStop print db close

* CV add changelog

* CV update changelog with attribution

(cherry picked from commit 48335a06f01524b036fde4dc1bab569bfc4ab9c7)

Co-authored-by: Chill Validation <[email protected]>
…lestiaorg#1214)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.25.0 to 1.25.1.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.25.0...v1.25.1)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…elestiaorg#1079) (celestiaorg#1222)

* Log proposer's address when correctly accepting a proposal (celestiaorg#1079)

* Log proposer when logging received proposal

* Addressed review comments

* Promote updates to validator to Info level

(cherry picked from commit cf230821c899f1f0cd3ef9cbbcc678ca01008d46)

# Conflicts:
#	consensus/state.go
#	state/execution.go

* Revert "Log proposer's address when correctly accepting a proposal (celestiaorg#1079)"

This reverts commit 8ccdb00e9023da91ae324c86956ffe5a5851d1a3.

* Log proposer's address when correctly accepting a proposal (celestiaorg#1079)

* Log proposer when logging received proposal

* Addressed review comments

* Promote updates to validator to Info level

---------

Co-authored-by: Sergio Mena <[email protected]>
…lestiaorg#1238)

Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.25.1 to 1.26.0.
- [Release notes](https://github.com/bufbuild/buf-setup-action/releases)
- [Commits](bufbuild/buf-setup-action@v1.25.1...v1.26.0)

---
updated-dependencies:
- dependency-name: bufbuild/buf-setup-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@rootulp
Copy link
Collaborator Author

rootulp commented Sep 25, 2024

This PR is blocked because unit tests haven't successfully run. The flaky tests are:

One idea is to avoid splitting the tests into random subgroups and instead run all the tests via one go test invocation. Based on experience in celestia-app that isn't any slower and often can be faster.

@rootulp rootulp marked this pull request as ready for review September 25, 2024 17:55
@rootulp rootulp requested a review from a team as a code owner September 25, 2024 17:55
@rootulp rootulp requested review from cmwaters and staheri14 and removed request for a team September 25, 2024 17:55
@rootulp
Copy link
Collaborator Author

rootulp commented Sep 25, 2024

CI is finally green 😌


"github.com/gogo/protobuf/proto"
)

const (
maxMsgSize = 104857600 // 100MB
maxMsgSize = math.MaxInt32 // 2GB
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know what this used for?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment on lines +739 to +753
// Experimental parameters to limit gossiping txs to up to the specified number of peers.
// This feature is only available for the default mempool (version config set to "v0").
// We use two independent upper values for persistent and non-persistent peers.
// Unconditional peers are not affected by this feature.
// If we are connected to more than the specified number of persistent peers, only send txs to
// ExperimentalMaxGossipConnectionsToPersistentPeers of them. If one of those
// persistent peers disconnects, activate another persistent peer.
// Similarly for non-persistent peers, with an upper limit of
// ExperimentalMaxGossipConnectionsToNonPersistentPeers.
// If set to 0, the feature is disabled for the corresponding group of peers, that is, the
// number of active connections to that group of peers is not bounded.
// For non-persistent peers, if enabled, a value of 10 is recommended based on experimental
// performance results using the default P2P configuration.
ExperimentalMaxGossipConnectionsToPersistentPeers int `mapstructure:"experimental_max_gossip_connections_to_persistent_peers"`
ExperimentalMaxGossipConnectionsToNonPersistentPeers int `mapstructure:"experimental_max_gossip_connections_to_non_persistent_peers"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I understand this correctly, this means that you can be connected to 20 peers but only gossip transactions (using the v0 mempool) for say 10 of those peers?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use the v0 mempool so do we still want these changes?

Copy link
Collaborator Author

@rootulp rootulp Sep 25, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't use the v0 mempool so do we still want these changes?

Node operators can still use the v0 mempool. FWIW I didn't selectively exclude any commits from v0.34.35. I pulled all upstream changes (including this one).

I don't see a strong reason to exclude this commit unless we're concerned it's dangerous or buggy. Now that you mention it, it does seem odd that an experimental feature was included in a patch release of CometBFT.

Copy link
Member

@evan-forbes evan-forbes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is always a ton of work, thanks a bunch @rootulp !!

mempool/v1/mempool_test.go Show resolved Hide resolved
Comment on lines -49 to +51
if len(v.Address) != crypto.AddressSize {
return fmt.Errorf("validator address is the wrong size: %v", v.Address)
addr := v.PubKey.Address()
if !bytes.Equal(v.Address, addr) {
return fmt.Errorf("validator address is incorrectly derived from pubkey. Exp: %v, got %v", addr, v.Address)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw to other reviewers, here's the fix with statesync

GHSA-g5xx-c4hv-9ccc

types/validator_set.go Show resolved Hide resolved
@rootulp rootulp merged commit 65c778d into celestiaorg:v0.34.x-celestia Sep 30, 2024
15 checks passed
@rootulp rootulp deleted the rp/v0.34.35 branch September 30, 2024 15:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

State syncing validator from malicious node may lead to a chain split