Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: blockbuilder component #14621

Draft
wants to merge 54 commits into
base: main
Choose a base branch
from
Draft

feat: blockbuilder component #14621

wants to merge 54 commits into from

Conversation

owen-d
Copy link
Member

@owen-d owen-d commented Oct 25, 2024

[WIP] Draft.

Introduces a "slimgester" component, which is essentially an ingester without write ahead logs, querying support (inc inverted indices or in-mem TSDB implementation), or limits application. It consumes a "job", aka(partition, offset_range) in kafka, writing our pre-existing format (chunks) to object storage. When finished, a TSDB is created from these chunks and written to storage.

This creates a way to idempotently consume a job specification and commit it in a more optimized form to a storage backend. We'll use this to iterate later on new storage formats, a scheduler+worker architecture for building, etc.

See plan.txt in this PR for more detail.

Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
Signed-off-by: Owen Diehl <[email protected]>
@owen-d owen-d changed the title RF1 Ingestion feat: blockbuilder component Nov 14, 2024
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Nov 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/XXL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant