Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better cleanup of output buckets after failed datums #36

Open
emk opened this issue Dec 2, 2022 · 0 comments
Open

Better cleanup of output buckets after failed datums #36

emk opened this issue Dec 2, 2022 · 0 comments

Comments

@emk
Copy link
Contributor

emk commented Dec 2, 2022

This is a follow-on to fixing #33.

From the source:

            // Remove `OutputFile` records for this datum, so we can upload the
            // same output files again.
            //
            // TODO: Unfortunately, there's an issue here. It takes one of two
            // forms:
            //
            // 1. Workers use deterministic file names. In this case, we
            //    _should_ be fine, because we'll just overwrite any files we
            //    did manage to upload.
            // 2. Workers use random filenames. Here, there are two subcases: a.
            //    We have successfully created an `OutputFile` record. b. We
            //    have yet to create an `OutputFile` record.
            //
            // We need to fix (2b) by pre-creating all our `OutputFile` records
            // _before_ uploading, and then updating them later to show that the
            // output succeeded. Which them into case (2a). And then we can fix (2a)
            // by deleting any S3/GCS files corresponding to `OutputFile::uri`.
emk added a commit that referenced this issue Dec 2, 2022
This code isn't quite perfect yet, but it's an improvement over current
beahvior.

Does not help with #36,
unfortunately.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant