Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Automatic task cleanup (without resumability) #4713

Closed
wants to merge 41 commits into from
Closed

Automatic task cleanup (without resumability) #4713

wants to merge 41 commits into from

Commits on Mar 27, 2023

  1. Add initial task graph and metadata json file 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 27, 2023
    Configuration menu
    Copy the full SHA
    47d0168 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2023

  1. Add task inputs and outputs to conrete DAG 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    ae67027 View commit details
    Browse the repository at this point in the history

  2. Fix failing tests 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 28, 2023
    Configuration menu
    Copy the full SHA
    8f95cd6 View commit details
    Browse the repository at this point in the history

Commits on Mar 29, 2023

  1. Use path-based APIs to get file metadata 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 29, 2023
    Configuration menu
    Copy the full SHA
    9f11e4b View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into ben-task-graph 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman authored Mar 29, 2023
    Configuration menu
    Copy the full SHA
    db6aed1 View commit details
    Browse the repository at this point in the history

Commits on Mar 30, 2023

  1. Use buffer to compute checksum 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 30, 2023
    Configuration menu
    Copy the full SHA
    8456892 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2023

  1. Add support for temporary output paths 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    77f2cdc View commit details
    Browse the repository at this point in the history

  2. Fix failing test 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    3e55ad5 View commit details
    Browse the repository at this point in the history

  3. Add caveat about overlapping output channels [ci skip] 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Mar 31, 2023
    Configuration menu
    Copy the full SHA
    e307f75 View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2023

  1. Delete files instead of emptying them (now supports directories and r… 

    …emote paths)

Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 1, 2023
    Configuration menu
    Copy the full SHA
    08881b0 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2023

  1. Add `eager' cleanup option 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    0cf07ec View commit details
    Browse the repository at this point in the history

  2. Fix concurrency issues [ci fast] 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 10, 2023
    Configuration menu
    Copy the full SHA
    73b2f3b View commit details
    Browse the repository at this point in the history

Commits on Apr 21, 2023

  1. Merge branch 'master' into ben-task-graph-pull

    @bentsherman
    bentsherman committed Apr 21, 2023
    Configuration menu
    Copy the full SHA
    0dd98d6 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2023

  1. Merge branch 'master' into ben-task-graph-pull 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    0f505d3 View commit details
    Browse the repository at this point in the history

  2. Replace synchronized with lock 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 26, 2023
    Configuration menu
    Copy the full SHA
    e81e584 View commit details
    Browse the repository at this point in the history

Commits on Apr 28, 2023

  1. Merge branch 'ben-task-graph-pull' into 452-eager-cleanup 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    ba2e7a6 View commit details
    Browse the repository at this point in the history

  2. Remove dependency on task graph branch 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    f7bcfa8 View commit details
    Browse the repository at this point in the history

  3. Use downstream tasks to determine lifetime for task cleanup 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Apr 28, 2023
    Configuration menu
    Copy the full SHA
    4d90e27 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. Rename TemporaryFileObserver to TaskCleanupObserver 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    dd23b2a View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into 452-eager-cleanup 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    6a34be6 View commit details
    Browse the repository at this point in the history

  3. Wait for output files to be published 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    ff08984 View commit details
    Browse the repository at this point in the history

  4. Log warning if eager cleanup is used with incompatible publish modes 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    9b343b6 View commit details
    Browse the repository at this point in the history

  5. Add eager cleanup for individual output files 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    6b5a820 View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. Add warning about includeInputs with eager cleanup 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    c42f249 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. Add resumability 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    8b66c39 View commit details
    Browse the repository at this point in the history

  2. Upgrade kryo to 5.4.0 (#3562) 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    43c939d View commit details
    Browse the repository at this point in the history

  3. Add serializer for HashCode 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    5a78cf9 View commit details
    Browse the repository at this point in the history

  4. Fix failing tests 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    7b02b8e View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. Revert Kryo upgrade 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    c727c05 View commit details
    Browse the repository at this point in the history

  2. Add kryo instantiator for HashCode 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    d37ee1f View commit details
    Browse the repository at this point in the history

  3. Remove HashCode serializer 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    859e96b View commit details
    Browse the repository at this point in the history

  4. Add log messages, fix infinite hang 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    c89811c View commit details
    Browse the repository at this point in the history

  5. Add support for includeInputs 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    a034f60 View commit details
    Browse the repository at this point in the history

  6. Update log messages 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    9741b1c View commit details
    Browse the repository at this point in the history

  7. Add lazy, eager, aggressive cleanup strategies 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    3aae1af View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2023

  1. Merge branch 'master' into 452-eager-cleanup 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Sep 13, 2023
    Configuration menu
    Copy the full SHA
    08342dd View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2023

  1. Add thread pool for task cleanup 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    3498c5d View commit details
    Browse the repository at this point in the history

  2. Minor improvement to checkCachedOutput() 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Sep 14, 2023
    Configuration menu
    Copy the full SHA
    c421617 View commit details
    Browse the repository at this point in the history

Commits on Jan 31, 2024

  1. Merge branch 'master' into 452-eager-cleanup 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Jan 31, 2024
    Configuration menu
    Copy the full SHA
    5b05aad View commit details
    Browse the repository at this point in the history

Commits on Feb 1, 2024

  1. minor edits 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    9ce10dc View commit details
    Browse the repository at this point in the history

  2. Remove WIP resumability 

    Signed-off-by: Ben Sherman <[email protected]>
    @bentsherman
    bentsherman committed Feb 1, 2024
    Configuration menu
    Copy the full SHA
    b73c6bb View commit details
    Browse the repository at this point in the history