compactor: concurrently get deletion marks for blocks

[dimitarvdimitrov]

What this PR does

With a lot of blocks churn (for example a lot of OOO blocks) getting the deletion marks for newly discovered blocks can take a long time (13 minutes).

This PR introduces changes the bucket index updater to request 16 deletions marks concurrently (same as the number of block deleted in parallel).

This is one example of a cleanup taking 20 minutes. 13 of those were in getting the deletion marks

started at 7:52
- {msg="marked blocks for deletion"}
	- very short (milliseconds?)
- {msg="skipped missing block deletion mark when updating bucket index"}
	- from 7:52 to 8:06
- {msg="listed all blocks in storage"}
	- very short
- {msg="updated deletion markers for recently marked blocks"}
	- very short
- {msg="skipped partial block when updating bucket index"}
	- first happened at 8:08; occurred until 8:10
- {msg="fetched blocks metas for newly discovered blocks"}
	- very short
- {msg="deleted block marked for deletion"}
	- from 8:10 to 8:13
- {msg="cleaned up partial blocks"}
- {msg="completed blocks cleanup and maintenance"}
- {msg="deleted partial block marked for deletion"}
	- all very short

ended at 8:13:19 (20m45.155959332s)

Which issue(s) this PR fixes or relates to

Fixes #

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• [n/a] about-versioning.md updated with experimental features.

[narqo]

🔥

[grafanabot]

The backport to r314 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-9881-to-r314 origin/r314
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 94255ac58ccfb68a82963d45ba912e17f3c784d0
# Push it to GitHub
git push --set-upstream origin backport-9881-to-r314
git switch main
# Remove the local backport branch
git branch -D backport-9881-to-r314

Then, create a pull request where the base branch is r314 and the compare/head branch is backport-9881-to-r314.

[grafanabot]

The backport to r314 failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new branch
git switch --create backport-9881-to-r314 origin/r314
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x 94255ac58ccfb68a82963d45ba912e17f3c784d0
# Push it to GitHub
git push --set-upstream origin backport-9881-to-r314
git switch main
# Remove the local backport branch
git branch -D backport-9881-to-r314

Then, create a pull request where the base branch is r314 and the compare/head branch is backport-9881-to-r314.