Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

More work on getting Dataflow to run #9

Merged
merged 67 commits into from
Aug 23, 2024
Merged

More work on getting Dataflow to run #9

merged 67 commits into from
Aug 23, 2024

Commits on Jul 12, 2024

  1. Fix recipe_id naming

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    71140d0 View commit details
    Browse the repository at this point in the history

  2. Update catalog.yaml

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    1ed8bba View commit details
    Browse the repository at this point in the history

  3. Update catalog.yaml

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    6c4f75d View commit details
    Browse the repository at this point in the history

  4. Update meta.yaml

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    cc99385 View commit details
    Browse the repository at this point in the history

  5. fix dependencies

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    d39ce90 View commit details
    Browse the repository at this point in the history

  6. bump pgf-recipes

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    66b4088 View commit details
    Browse the repository at this point in the history

  7. Update config_dataflow.py

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    6816b1a View commit details
    Browse the repository at this point in the history

  8. move final output to persistent bucket

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    92102ab View commit details
    Browse the repository at this point in the history

  9. Re add logic for copy stage

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    363fb00 View commit details
    Browse the repository at this point in the history

  10. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    3c183f1 View commit details
    Browse the repository at this point in the history

  11. remove prune and reduce machine size

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    e9ae02d View commit details
    Browse the repository at this point in the history

  12. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    b19e7a3 View commit details
    Browse the repository at this point in the history

  13. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    6152eb6 View commit details
    Browse the repository at this point in the history

  14. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    57d68ed View commit details
    Browse the repository at this point in the history

  15. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    9e27af0 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. Update config_dataflow.py

    @jbusecke
    jbusecke committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    4f89453 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2024

  1. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    4258766 View commit details
    Browse the repository at this point in the history

  2. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    de123d0 View commit details
    Browse the repository at this point in the history

  3. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    558b7c0 View commit details
    Browse the repository at this point in the history

  4. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    ec2da10 View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    49728c3 View commit details
    Browse the repository at this point in the history

  6. Update eNATL60.py

    @jbusecke
    jbusecke committed Jul 23, 2024
    Configuration menu
    Copy the full SHA
    4f64075 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2024

  1. converts cftime.DatetimeGregorian datetime64[ns]

    @norlandrhagen
    norlandrhagen committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    dc648fd View commit details
    Browse the repository at this point in the history

  2. pruned + job name

    @norlandrhagen
    norlandrhagen committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    d3c8464 View commit details
    Browse the repository at this point in the history

  3. adds copy stage & echo

    @norlandrhagen
    norlandrhagen committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    09ae870 View commit details
    Browse the repository at this point in the history

  4. noprune

    @norlandrhagen
    norlandrhagen committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    f60775c View commit details
    Browse the repository at this point in the history

  5. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Aug 8, 2024
    Configuration menu
    Copy the full SHA
    da43ea3 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2024

  1. hardcode dataflow name + prune

    @norlandrhagen
    norlandrhagen committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    d9cf730 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'recipe-id-wo-underscore' of https://github.com/leap-stc… 

    …/eNATL_feedstock into recipe-id-wo-underscore
    @norlandrhagen
    norlandrhagen committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    04c119b View commit details
    Browse the repository at this point in the history

  3. 1 month + prod copy

    @norlandrhagen
    norlandrhagen committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    6c592b7 View commit details
    Browse the repository at this point in the history

  4. hardcode naming

    @norlandrhagen
    norlandrhagen committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    15b4ad9 View commit details
    Browse the repository at this point in the history

  5. naming nit

    @norlandrhagen
    norlandrhagen committed Aug 9, 2024
    Configuration menu
    Copy the full SHA
    f9d03e8 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. max_concurrency=1 & prune off

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    72861b2 View commit details
    Browse the repository at this point in the history

  2. prune bool - does this work

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    023d13a View commit details
    Browse the repository at this point in the history

  3. disable prime, add e2-highmem-4

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    47ecf4c View commit details
    Browse the repository at this point in the history

  4. bump machine size

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    8ef929f View commit details
    Browse the repository at this point in the history

  5. new approach

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4fb61ff View commit details
    Browse the repository at this point in the history

  6. [pre-commit.ci] auto fixes from pre-commit.com hooks 

    for more information, see https://pre-commit.ci
    @pre-commit-ci
    pre-commit-ci[bot] committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    aff5758 View commit details
    Browse the repository at this point in the history

  7. lint

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    84caede View commit details
    Browse the repository at this point in the history

  8. Merge branch 'recipe-id-wo-underscore' of https://github.com/leap-stc… 

    …/eNATL_feedstock into recipe-id-wo-underscore
    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    55a8a60 View commit details
    Browse the repository at this point in the history

  9. pooch

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    39bfabe View commit details
    Browse the repository at this point in the history

  10. pooch 5 timesteps

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    8ce62e1 View commit details
    Browse the repository at this point in the history

  11. update reqs

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4cb16c9 View commit details
    Browse the repository at this point in the history

  12. 5 to 32 days

    @norlandrhagen
    norlandrhagen committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4b18288 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. smaller worker

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    ee1dbaa View commit details
    Browse the repository at this point in the history

  2. trying out ARM machine

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    7159713 View commit details
    Browse the repository at this point in the history

  3. small standard worker

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    b7d28e3 View commit details
    Browse the repository at this point in the history

  4. single worker + highmem

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    23a37a9 View commit details
    Browse the repository at this point in the history

  5. task

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    4ddfd92 View commit details
    Browse the repository at this point in the history

  6. disk size

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    c529398 View commit details
    Browse the repository at this point in the history

  7. 300GB storage

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    5c41c75 View commit details
    Browse the repository at this point in the history

  8. 120 days

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    95211e4 View commit details
    Browse the repository at this point in the history

  9. weird z3 machine

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    39b7d1f View commit details
    Browse the repository at this point in the history

  10. region for z3

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    642af62 View commit details
    Browse the repository at this point in the history

  11. use_shuffle = False + fork of runner with new opts

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    1f91f7a View commit details
    Browse the repository at this point in the history

  12. nit

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    c2c8b80 View commit details
    Browse the repository at this point in the history

  13. max_num_workers=1

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    26baa53 View commit details
    Browse the repository at this point in the history

  14. tmask

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    dcf3066 View commit details
    Browse the repository at this point in the history

  15. shuffle off retry + 34 days

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    aa09a95 View commit details
    Browse the repository at this point in the history

  16. nit

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    ff79c11 View commit details
    Browse the repository at this point in the history

  17. pruned version, no copy w/ shuffle and ssd

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    b9d9ea7 View commit details
    Browse the repository at this point in the history

  18. eNATL full recipe + shuffle no copy

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    99af8c6 View commit details
    Browse the repository at this point in the history

  19. pooch concurrent test

    @norlandrhagen
    norlandrhagen committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    53c6683 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. coord update

    @norlandrhagen
    norlandrhagen committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    231d1de View commit details
    Browse the repository at this point in the history

  2. 1 worker

    @norlandrhagen
    norlandrhagen committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    8f3b264 View commit details
    Browse the repository at this point in the history

  3. config update

    @norlandrhagen
    norlandrhagen committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    5d32583 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. incorporate review + smaller worker + prune

    @norlandrhagen
    norlandrhagen committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    1c33507 View commit details
    Browse the repository at this point in the history