Skip to content

Commit

Permalink
restore: waive 200 file limit for online restores
Browse files Browse the repository at this point in the history
Online restore wants big/huge chunks of spans passed to its workers so
that they spend the most time splitting and linking inside a chunk and
less time splitting chunks away from other workers. This is why we bump
the target size to 16gb, but we also don't want a 200 file limit getting
in the way of hitting this size.

Release note: none.
Epic: none.
  • Loading branch information
dt committed Mar 29, 2024
1 parent 0dce294 commit e9db14a
Showing 1 changed file with 5 additions and 1 deletion.
6 changes: 5 additions & 1 deletion pkg/ccl/backupccl/restore_span_covering.go
Original file line number Diff line number Diff line change
Expand Up @@ -433,7 +433,11 @@ func generateAndSendImportSpans(
// we want to bound sum total open files across all of them to <= 1k.
// We bound the span byte size to improve work distribution and make
// the progress more granular.
fits := lastCovSpanSize+newCovFilesSize <= filter.targetSize && lastCovSpanCount+newCovFilesCount <= 200
// We waive the file count limit if the target size is >8GB, as a size
// this large clearly indicates we want lots of files (such as during
// online restores where we don't open them).
fileCountOK := lastCovSpanCount+newCovFilesCount <= 200 || filter.targetSize > 8<<30
fits := lastCovSpanSize+newCovFilesSize <= filter.targetSize && fileCountOK

if (newCovFilesCount == 0 || fits) && !firstInSpan {
// If there are no new files that cover this span or if we can add the
Expand Down

0 comments on commit e9db14a

Please sign in to comment.