Parallelize backup verification

[mszeszko-meta]

Summary

Today, backup verification is serial, which could pose a challenge in rare, high urgency recovery scenarios where we want to timely assess whether candidate backup is not corrupted and eligible for the restore. The timely part will become increasingly more important in case of disaggregated storage.

Semantics

Given the very simple thread pool implementation in backup_engine today, we do not really have a control over initialized threads and consequently do not have an option to unschedule / cancel in-progress tasks. As a result, VerifyBackup won't bail out on a very first mismatch (as it was the case for serial implementation) and instead will iterate over all the files logging success / degree_of_failure for each. We could, in theory, not .wait() on remaining std::future<WorkItem>s (upon previously detected failure) and therefore decrease the observed API latency, but that could cause more confusion down the road as verification threads would still be occupied with inflight/scheduled work and would not be reclaimed by the pool for a while. It's a tradeoff where we choose a solution with clear and intuitive semantics.

Test plan

Make sure existing test collateral for backup verification in BackupEngineTest (VerifyBackup,CorruptFileMaintainSize,CorruptBlobFileMaintainSize, TableFileCorruptionBeforeIncremental, RateLimitingVerifyBackup, RateLimitingWithLowRefillBytesPerPeriod,Concurrency) runs in db_stress.

[facebook-github-bot]

@mszeszko-meta has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.