Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WAL/Journal-like bucket #255

Open
ransomw1c opened this issue Sep 5, 2019 · 1 comment
Open

WAL/Journal-like bucket #255

ransomw1c opened this issue Sep 5, 2019 · 1 comment
Labels
spike Design, investigation tasks or tasks that does not have clear deliverable

Comments

@ransomw1c
Copy link
Contributor

as part of the plans for accelerator pods (to support either search- or query-like functionality, more details anon), we need to allow ~/.datamon/datamon.yaml to ref an additional bucket. this bucket will be used to facet searches, journal writes, etc. by pipeline- or otherwise project-specific in-cloud accelerators.

for the scope of this task, we're essentially talking about adding journaling (in the sense of journalled filesystems) or Write Ahead Logging (in the sense of Postgres) to commands that write to existing (meta and blob) stores.

@ransomw1c ransomw1c added the spike Design, investigation tasks or tasks that does not have clear deliverable label Sep 8, 2020
@ransomw1c
Copy link
Contributor Author

the proposal and client-side WAL implementation are progress on this iss.

these items do not cover this issue because this issue is envisioning additional golang to not just upload to a context (in the datamon sense, post WAL implementation, not the generalized golang sense, a context mostly characterized by a set of [AWS/gg] buckets) that contains a WAL but also to migrate data between two contexts according to some specialized rules. in general, this would effectively allow end-user functionalities like incremental uploads. i've added the "spike" label to indicate that the exact functionalities for transfer between contexts – much less mapping these 'machine level instructions' (i.e. the scope of the spike) to 'productizeable features' (e.g. incremental uploads).

at time of writing, i opine features such as progress bars and others that can be realized without multiple blob buckets (or multiple contexts, whichever requirement rings truer – they are used interchangeably here) as described #443 are merely papering over the engineering work, something like very simple GC, essentially, described in this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
spike Design, investigation tasks or tasks that does not have clear deliverable
Development

No branches or pull requests

1 participant