Problem: memiavl is not benchmarked against realworld data

[yihuang]

Open this issue just to save the local benchmark script and result, it can't be run in CI because one need to prepare the data first.

The Benchmark Code

func BenchmarkRealWorld(b *testing.B) {
	firstKey, err := hex.DecodeString("0100040776cd54b3166e6db36e534d6a1fff161d20b443597dc44f10d9ad23f049")
	require.NoError(b, err)
	expValueLen := 2175

	mtree, err := LoadMultiTree("/path/to/.cronos/data/memiavl.db/snapshot-7920000")
	require.NoError(b, err)
	b.Run("memiavl", func(b *testing.B) {
		tree := mtree.TreeByName("evm")
		// warm up
		v := tree.Get(firstKey)
		require.Equal(b, expValueLen, len(v))
		b.ResetTimer()
		for i := 0; i < b.N; i++ {
			_ = tree.Get(firstKey)
		}
	})

	db, err := dbm.NewDB("application", dbm.RocksDBBackend, "/path/to/.cronos/data")
	require.NoError(b, err)
	defer db.Close()
	prefix := "s/k:evm/"
	db = dbm.NewPrefixDB(db, []byte(prefix))
	tree2, err := iavl.NewMutableTreeWithOpts(db, 0, nil, false)
	require.NoError(b, err)
	_, err = tree2.LoadVersion(7920000)
	require.NoError(b, err)
	b.Run("iavl-fastnode", func(b *testing.B) {
		// warm up
		v, err := tree2.Get(firstKey)
		require.NoError(b, err)
		require.Equal(b, expValueLen, len(v))
		b.ResetTimer()
		for i := 0; i < b.N; i++ {
			_, _ = tree2.Get(firstKey)
		}
	})

	tree3, err := iavl.NewMutableTreeWithOpts(db, 0, nil, true)
	require.NoError(b, err)
	_, err = tree3.LoadVersion(7920000)
	require.NoError(b, err)
	b.Run("iavl", func(b *testing.B) {
		// warm up
		v, err := tree3.Get(firstKey)
		require.NoError(b, err)
		require.Equal(b, expValueLen, len(v))
		b.ResetTimer()
		for i := 0; i < b.N; i++ {
			_, _ = tree3.Get(firstKey)
		}
	})
}

Prepare Data

Prepare the memiavl.db and application.db for snapshot at Cronos mainnet height 7920000:

• Run state sync with memiavl will restore the memiavl.db, after completed, remove the wal directory to keep it clean.
• Run cronosd changeset restore-app-db command to convert memiavl.db/snapshot-7920000 to rocksdb application.db.
• build fast node index on application.db:

    package main
  
  import (
    "github.com/cosmos/iavl"
    dbm "github.com/tendermint/tm-db"
  )
  
  func main() {
    db, err := dbm.NewDB("application", dbm.RocksDBBackend, "/path/to/.cronos/data")
    if err != nil {
    	panic(err)
    }
    defer db.Close()
    prefix := "s/k:evm/"
    db = dbm.NewPrefixDB(db, []byte(prefix))
    tree, err := iavl.NewMutableTreeWithOpts(db, 500000, nil, false)
    if err != nil {
    	panic(err)
    }
    _, err = tree.LoadVersion(7920000)
    if err != nil {
    	panic(err)
    }
  }

[yihuang]

Benchmark result:

goos: darwin
goarch: amd64
pkg: github.com/crypto-org-chain/cronos/memiavl
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkRealWorld/memiavl-12      4701506    249.4 ns/op    0 B/op    0 allocs/op
BenchmarkRealWorld/memiavl-12      4670912    257.2 ns/op    0 B/op    0 allocs/op
BenchmarkRealWorld/memiavl-12      4664716    256.2 ns/op    0 B/op    0 allocs/op
BenchmarkRealWorld/iavl-fastnode-12   304472   3626 ns/op  4880 B/op   12 allocs/op
BenchmarkRealWorld/iavl-fastnode-12   313591   3632 ns/op  4880 B/op   12 allocs/op
BenchmarkRealWorld/iavl-fastnode-12   320437   3650 ns/op  4880 B/op   12 allocs/op
BenchmarkRealWorld/iavl-12       8084  148257 ns/op  20192 B/op   363 allocs/op
BenchmarkRealWorld/iavl-12       8041  144762 ns/op  20192 B/op   363 allocs/op
BenchmarkRealWorld/iavl-12       8070  144806 ns/op  20192 B/op   363 allocs/op
PASS
ok  github.com/crypto-org-chain/cronos/memiavl 13.033s

[yihuang]

BTW, sync with local blocks from height 7920001 to 7923597 takes 2minutes with memiavl, 7minutes with iavl.

So it's like 3times end-to-end performance improvements, there could be more improvements after optimizations on cache store.

EDIT: with async wal writing, the block replay time is further reduced from 2m to 1m10s.
That's 3597 / 70 = 51 blocks per second, with that speed on average, sync the mainnet from genesis only takes 8029283 / 51 / 3600 / 24 = 1.82 days

[yihuang]

More performance data collected from local runs on my laptop against mainnet snapshot.

	Time	Peak RSS
Load snapshot 7920000 and replay WAL to 7928042.	12s	2.38G
Load snapshot 7920000 and replay WAL to 7928042 and do a snapshot rewrite.	4m14s	14.54G

• snapshot is from cronos mainnet
• data is reported by linux time utility, time --verbose, (the builtin one in mac is different, need to install manually).
• snapshot size 14G.
• the WAL size of 8042 blocks is 194M.

The data seems suggest that it's fine to set snapshot interval to thousands of blocks, without slowdown startup too much, or increase too much memory usage. For different chains, it depends on the traffic on the chain.

The golang program for this:

package main

import (
	"fmt"

	"github.com/crypto-org-chain/cronos/memiavl"
)

func main() {
	db, err := memiavl.Load("/path/to/.cronos/data/memiavl.db", memiavl.Options{})
	if err != nil {
		panic(err)
	}

	fmt.Println("loaded", db.Version())

	if err := db.RewriteSnapshot(); err != nil {
		panic(err)
	}

	if err := db.Reload(); err != nil {
		panic(err)
	}

	fmt.Println("rewrited", db.Version())
}

[yihuang]

https://github.com/crypto-org-chain/cronos/wiki/MemIAVL-Benchmark

recent benchmarks.