Notional tracking issue

[faddat]

This is tracking work that notional is doing, and summarizing a set of recommendations around databases in tendermint and cosmos. Little check-mark emojis will be used to indicate done-ness.

#245 - Rocksdb folder creation ✅ , should be merged so that Gaia and others can state sync. Osmosis state sync is blocked here:

osmosis-labs/osmosis#1381

#246 ❌ - Boltdb folder creation when we do it so it passes tests here, Gaia fails to start with it. @creachadair Suggests standardizing folder creation & setup.

#239 ✅ - Vuong will implement changes so that tests don't only use memdb. Closes #156.

#237 - Jacob is using this as a reference branch and is running it in production on evmos, where the notional node is outperforming the nearest by 28x (missed blocks). Picture is worth a thousand words:

#204 ✅ - upgrading from gorocksdb to grocksdb allows us to use a current version of rocksdb

Resource constraints pose an actual risk of failure to downstream chains. We should be recommending that they use the latest version of this library, so that we can capture the data that we need.

@marbar3778 mentioned creating a repository under the cosmos org where we can host the build for the docker image containing the current version of rocksdb (and literally all other cosmos tooling)

#186 ✅ ❓ : Implement item on itself not only its pointer - @ValarDragon has noticed some room for performance improvements.

#188 ❌ - @ValarDragon has noticed some room for performance improvements, but this seems to not be as high priority.

medium term

Osmosis, kava, evmos, Juno, and other resource constrained chains should move to sdk 46 and tendermint 35. Since tendermint 35 and sdk 46 still import this library, keeping it up to date remains critical to understanding performance in cosmos / tendermint.

We need real world performance comparisons once that’s happened, currently most data (including my own) is conjecture and “feeling”.

Tendermint team wants to standardize on badgerdb.

cosmos-sdk team has chosen badger and cosmos/gorocksdb. Jacob suggests that we stop maintaining an independent rocksdb library, and use grocksdb. Alternatively, we can keep cosmos/gorocksdb up to date with upstream grocksdb, but I think that it is best that we stop maintaining that.

@marbar3778 says that we should look to ethereum for solutions, and that their tests indicate that using goleveldb is best long run, but that we need to use it quite differently. Ethereum does use it quite differently.

long term

Everyone seems to be in favor of specialization— that is, for things like the sdk and tendermint, choosing a single db, and using it idiomatically. Everyone seems to agree that we should drop support for databases that use cgo.

Jacob is concerned about badger. The limitations on value sizes in badger make impossible to use large genesises, like those that would be necessary for a genesis export upgrade.

Upstream, code has been written for badger that solves this, but the badger team did not merge or respond to it.

The kv stores of interest would then be:

• badger -
  • concern: enlarge maxValueThreshold dgraph-io/badger#1754
    • This prevents evmos from using badger: the limitation on value sizes stops it.
• goleveldb
  • concern: not very active
• pebble
  • concern: @creachadair reports that its attempt to be rocks comptabile can harm its performance and long term viability
  • pro: fast moving code base
• bolt
  • Stable, terra team reports that it does best with reads.

Of these, it seems that pebble is getting the most active support from its team.

It is probably worth noting that there's no need for the sdk and tendermint to be using the same database... except maybe this can help us to save some work.