Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

record: allow unbounded queued blocks in LogWriter #2762

Merged
merged 2 commits into from
Jul 24, 2023
Merged

record: allow unbounded queued blocks in LogWriter #2762

merged 2 commits into from
Jul 24, 2023

Commits on Jul 24, 2023

  1. record: allow unbounded queued blocks in LogWriter 

    Previously, a LogWriter would allocate up to 16 blocks of 32 KiB for buffering
WAL writes. If all 16 blocks had been allocated and no free blocks were
available, a batch writing to the WAL would queue until the flushing goroutine
freed blocks. In testing of write-heavy workloads, especially with larger value
sizes, we've seen queueing at the LogWriter. This queueing blocks the commit
pipeline, preventing any batches from committing regardless of priority and
whether they require waiting for fsync.

This commit modifies LogWriter to allow the queueing of an unbounded number of
blocks. In practice, for the current WAL, the memtable size serves as an upper
bound. With a 64 MiB memtable, at most 64 MiB / 32 KiB = 2,048 blocks may
queue. This is not an unreasonable of additional memory overhead for a
write-heavy workload.

Beyond improving throughput for write-heavy workloads, removing this hard bound
improves tolerance of momentary disk stalls.

Informs cockroachdb/cockroach#88699.
    @jbowens
    jbowens committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    bb67bab View commit details
    Browse the repository at this point in the history

  2. record: pool LogWriter blocks 

    When a LogWriter is closed, return its blocks to a sync.Pool for reuse by a
subsequent LogWriter to reduce allocations.

```
goos: darwin
goarch: arm64
pkg: github.com/cockroachdb/pebble/record
                                    │   old.txt    │               new.txt               │
                                    │    sec/op    │   sec/op     vs base                │
QueueWALBlocks/record-size=64B-10      32.80m ± 2%   28.23m ± 2%  -13.93% (p=0.000 n=10)
QueueWALBlocks/record-size=512B-10     15.12m ± 2%   11.55m ± 1%  -23.58% (p=0.000 n=10)
QueueWALBlocks/record-size=1.0KB-10    14.23m ± 4%   10.93m ± 4%  -23.20% (p=0.000 n=10)
QueueWALBlocks/record-size=2.0KB-10    13.84m ± 2%   10.52m ± 2%  -24.00% (p=0.000 n=10)
QueueWALBlocks/record-size=32KB-10    13.197m ± 1%   9.784m ± 2%  -25.87% (p=0.000 n=10)
geomean                                16.67m        12.97m       -22.22%

                                    │   old.txt    │               new.txt                │
                                    │     B/s      │     B/s       vs base                │
QueueWALBlocks/record-size=64B-10     1.906Gi ± 2%   2.214Gi ± 2%  +16.18% (p=0.000 n=10)
QueueWALBlocks/record-size=512B-10    4.134Gi ± 2%   5.411Gi ± 1%  +30.86% (p=0.000 n=10)
QueueWALBlocks/record-size=1.0KB-10   4.393Gi ± 4%   5.721Gi ± 4%  +30.21% (p=0.000 n=10)
QueueWALBlocks/record-size=2.0KB-10   4.514Gi ± 2%   5.940Gi ± 2%  +31.58% (p=0.000 n=10)
QueueWALBlocks/record-size=32KB-10    4.736Gi ± 1%   6.388Gi ± 2%  +34.89% (p=0.000 n=10)
geomean                               3.748Gi        4.820Gi       +28.58%

                                    │     old.txt     │               new.txt                │
                                    │      B/op       │     B/op      vs base                │
QueueWALBlocks/record-size=64B-10     96058.79Ki ± 0%   99.72Ki ± 1%  -99.90% (p=0.000 n=10)
QueueWALBlocks/record-size=512B-10    83738.81Ki ± 0%   98.77Ki ± 0%  -99.88% (p=0.000 n=10)
QueueWALBlocks/record-size=1.0KB-10   82858.79Ki ± 0%   98.99Ki ± 0%  -99.88% (p=0.000 n=10)
QueueWALBlocks/record-size=2.0KB-10   82418.81Ki ± 0%   98.78Ki ± 0%  -99.88% (p=0.000 n=10)
QueueWALBlocks/record-size=32KB-10    82019.15Ki ± 0%   99.02Ki ± 0%  -99.88% (p=0.000 n=10)
geomean                                  83.26Mi        99.06Ki       -99.88%

                                    │   old.txt    │              new.txt               │
                                    │  allocs/op   │ allocs/op   vs base                │
QueueWALBlocks/record-size=64B-10     2413.00 ± 0%   14.00 ± 0%  -99.42% (p=0.000 n=10)
QueueWALBlocks/record-size=512B-10    2105.00 ± 0%   14.00 ± 0%  -99.33% (p=0.000 n=10)
QueueWALBlocks/record-size=1.0KB-10   2083.00 ± 0%   14.00 ± 7%  -99.33% (p=0.000 n=10)
QueueWALBlocks/record-size=2.0KB-10   2072.00 ± 0%   14.00 ± 0%  -99.32% (p=0.000 n=10)
QueueWALBlocks/record-size=32KB-10    2062.00 ± 0%   14.00 ± 7%  -99.32% (p=0.000 n=10)
geomean                                2.143k        14.00       -99.35%
```
    @jbowens
    jbowens committed Jul 24, 2023
    Configuration menu
    Copy the full SHA
    4445b51 View commit details
    Browse the repository at this point in the history