Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add io_buffer_size to BackupEngineOptions #13236

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

archang19
Copy link
Contributor

@archang19 archang19 commented Dec 19, 2024

Summary

The RocksDB backup engine code currently derives the IO buffer size based on the following criteria:

  • If specified, use the rate limiter burst size
  • Otherwise, use the default size (5 MiB)

We want to be able to explicitly choose the IO size based on the storage backend. We want the new criteria to be:

  • If specified, use the size in BackupEngineOptions
  • If specified, use the rate limiter burst size
  • Otherwise, use the default size (5 MiB)

This PR adds a new option called io_buffer_size to BackupEngineOptions and updates the logic used to set the buffer size.

Test Plan

I added a separate unit test and verified that we can either use the io_buffer_size, rate limiter burst size, or the default size.

I decided to use a TEST_SYNC_POINT_CALLBACK. I considered the alternative of updating the Read implementation of DummySequentialFile / CheckIOOptsSequentialFile to check the value of n. However, that would have considerably complicated the whole test code, and we also do not need to be checking for this in every single test case. I think the TEST_SYNC_POINT_CALLBACK turned out to be quite elegant.

@archang19 archang19 force-pushed the backup-engine-io-buffer-size branch 5 times, most recently from da39170 to 1316c31 Compare December 20, 2024 23:13
@archang19 archang19 force-pushed the backup-engine-io-buffer-size branch from 1316c31 to 82348e9 Compare December 20, 2024 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants