Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[merkledb benchmark] implement simple write profile benchmark #3372

Open
wants to merge 44 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
da7f632
update
tsachiherman Sep 8, 2024
67b83d9
update
tsachiherman Sep 8, 2024
60a1bd7
update
tsachiherman Sep 8, 2024
e9f6f20
registration order.
tsachiherman Sep 8, 2024
2ce08b8
add few more metrics.
tsachiherman Sep 8, 2024
a8b673f
update
tsachiherman Sep 8, 2024
e888f9e
fix metric
tsachiherman Sep 8, 2024
a34e949
update
tsachiherman Sep 8, 2024
8984569
few more updates.
tsachiherman Sep 9, 2024
c844c70
reuse batch
tsachiherman Sep 9, 2024
30f05e7
update
tsachiherman Sep 9, 2024
e2ed004
bugfix
tsachiherman Sep 9, 2024
24da31d
avoid double registration
tsachiherman Sep 10, 2024
171c3b5
add leveldb configuration; optimize writes.
tsachiherman Sep 10, 2024
3d9fd61
Merge branch 'master' into tsachi/bench_merkledb
tsachiherman Sep 10, 2024
c16e891
update
tsachiherman Sep 10, 2024
bfe6c7c
update
tsachiherman Sep 10, 2024
4a0ae2b
update per CR.
tsachiherman Sep 12, 2024
a4d935f
Merge branch 'master' into tsachi/bench_merkledb
tsachiherman Sep 12, 2024
2014fdc
add a benchmark to compare with eth's merkle trie.
tsachiherman Sep 16, 2024
af5c4ab
udpate
tsachiherman Sep 16, 2024
4f807c8
update caches
tsachiherman Sep 18, 2024
291f2da
fix bench
tsachiherman Sep 18, 2024
058ec55
fix typo
tsachiherman Sep 18, 2024
0c80953
update dashboard.
tsachiherman Sep 19, 2024
a6385b0
update
tsachiherman Sep 20, 2024
6a1817e
update
tsachiherman Sep 20, 2024
716fbbd
refactor and add few unit test for database creation.
tsachiherman Sep 22, 2024
dd25de9
add unit test.
tsachiherman Sep 22, 2024
ae64496
update revisions
tsachiherman Sep 23, 2024
f3191bc
adding historical lookup for geth.
tsachiherman Sep 23, 2024
a0f31a4
Fix geth merkledb unit test (#3414)
StephenButtolph Sep 25, 2024
4602deb
Code cleanups (#3416)
rkuris Sep 26, 2024
72b9d13
fix lint
tsachiherman Sep 26, 2024
b94e8f3
update
tsachiherman Sep 28, 2024
2485f98
update
tsachiherman Sep 28, 2024
b2aa009
update
tsachiherman Sep 28, 2024
ceada16
update
tsachiherman Sep 28, 2024
9ff9dbe
update setup
tsachiherman Sep 28, 2024
ddd0c88
updatr
tsachiherman Sep 28, 2024
1b87ea8
use less memory for cache so that it would fit for 16gb machines.
tsachiherman Sep 29, 2024
78d164b
fix linter issues
tsachiherman Sep 29, 2024
e3a0002
further reduce cache size
tsachiherman Sep 29, 2024
d9327ae
update
tsachiherman Oct 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
407 changes: 407 additions & 0 deletions x/merkledb/benchmarks/benchmark.go

Large diffs are not rendered by default.

66 changes: 66 additions & 0 deletions x/merkledb/benchmarks/benchmark_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
// Copyright (C) 2019-2024, Ava Labs, Inc. All rights reserved.
// See the file LICENSE for licensing terms.

package main

import (
"context"
"os"
"testing"

"github.com/prometheus/client_golang/prometheus"
"github.com/stretchr/testify/require"

"github.com/ava-labs/avalanchego/database"
"github.com/ava-labs/avalanchego/database/leveldb"
"github.com/ava-labs/avalanchego/utils/logging"
"github.com/ava-labs/avalanchego/x/merkledb"
)

func TestGoldenDatabaseCreation(t *testing.T) {
for _, size := range []uint64{9500, 20000, 50000, 103011} {
testGoldenDatabaseCreation(t, size)
}
}

func testGoldenDatabaseCreation(t *testing.T, size uint64) {
path := getGoldenDatabaseDirectory(size)
if Exists(getGoldenDatabaseDirectory(size)) {
require.NoError(t, os.RemoveAll(path))
}
require.NoError(t, createGoldenDatabase(size))
require.NoError(t, resetRunningDatabaseDirectory(size))
testGoldenDatabaseContent(t, size)
require.NoError(t, os.RemoveAll(path))
require.NoError(t, os.RemoveAll(getRunningDatabaseDirectory(size)))
}

func testGoldenDatabaseContent(t *testing.T, size uint64) {
levelDB, err := leveldb.New(
getRunningDatabaseDirectory(size),
getLevelDBConfig(),
logging.NoLog{},
prometheus.NewRegistry(),
)
require.NoError(t, err)
mdb, err := merkledb.New(context.Background(), levelDB, getMerkleDBConfig(nil))
require.NoError(t, err)
defer func() {
mdb.Close()
levelDB.Close()
}()
// read values and make sure they are what we expect them to be.
for keyIndex := uint64(0); keyIndex < size; keyIndex++ {
entryHash := calculateIndexEncoding(keyIndex)
entryValue, err := mdb.Get(entryHash)
require.NoError(t, err)
require.Equal(t, entryHash, entryValue)
}

// try an entry beyond.
entryHash := calculateIndexEncoding(size + 1)
_, err = mdb.Get(entryHash)
require.ErrorIs(t, err, database.ErrNotFound)

require.NoError(t, mdb.Clear())
}
Loading
Loading