Skip to content

Commit

Permalink
fix: enabling badger gc (#2020)
Browse files Browse the repository at this point in the history
While profiling with pyroscope, I noticed that while
`badger/v2.(*DB).doWrites` stays constant for days,
`badger/v2/y.(*WaterMark)` grows consistently over days.

`Watermark` helps in managing the memory usage and ensuring that certain
thresholds are not exceeded, allowing the system to perform
optimizations, such as compaction. Badger's garbage collection (GC)
primarily targets the value log to free up space on disk, not in memory.
The value log GC reclaims space by removing stale and obsolete data from
the value log files, which are stored on disk. BUT: invoking the
RunValueLogGC() method can have an _indirect_ impact on memory usage:
value log GC occurs helps stabilize the growth of the WaterMark by
preventing an excessive number of small value log files from
accumulating. This can, in turn, lead to more efficient memory usage, as
the LSM tree components (MemTable and SSTables) do not need to track as
many small value log files.
  • Loading branch information
distractedm1nd authored Apr 5, 2023
1 parent f480a8f commit 8e36597
Showing 1 changed file with 0 additions and 4 deletions.
4 changes: 0 additions & 4 deletions nodebuilder/store.go
Original file line number Diff line number Diff line change
Expand Up @@ -144,10 +144,6 @@ func (f *fsStore) Datastore() (_ datastore.Batching, err error) {
// Bigger values constantly takes more RAM
// TODO(@Wondertan): Make configurable with more conservative defaults for Light Node
opts.MaxTableSize = 64 << 20
// Remove GC as long as we don't have pruning of data to be GCed.
// Currently, we only append data on disk without removing.
// TODO(@Wondertan): Find good enough default, once pruning is shipped.
opts.GcInterval = 0

f.data, err = dsbadger.NewDatastore(dataPath(f.path), &opts)
if err != nil {
Expand Down

0 comments on commit 8e36597

Please sign in to comment.