Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature: Sim mode at runtime #5721

Merged
merged 52 commits into from
Mar 15, 2024
Merged

Feature: Sim mode at runtime #5721

merged 52 commits into from
Mar 15, 2024

Commits on Feb 28, 2024

  1. wip on simulation broadcastable

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    7c4c1b9 View commit details
    Browse the repository at this point in the history

  2. Allow Broadcasting modification of sim mode tasks 

    * Create a mode	settings object - contains mode	run info.
* Test:	Sim time check retrieves config	on restart.
* Make custom outputs happen on	failure	as well	as success.
* Check	for broadcasts at simulation job submit.

Apply suggestions from code review

Co-authored-by: Oliver Sanders <[email protected]>

Response to review
Removed get_simulation_time from submit time.

response to review

save and retreive simulation task start time across reloads

make sim mode obey queued/runahead/held status

Update cylc/flow/simulation.py

Co-authored-by: Ronnie Dutta <[email protected]>

Apply suggestions from code review

Co-authored-by: Ronnie Dutta <[email protected]>

Update tests/integration/test_simulation.py

Co-authored-by: Ronnie Dutta <[email protected]>

Update tests/integration/test_simulation.py

Co-authored-by: Ronnie Dutta <[email protected]>

response to review

simplified ModeSettings Usage

Apply suggestions from code review

Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    c5f3a25 View commit details
    Browse the repository at this point in the history

  3. return to get_simulation. Cache call to time()

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    6f5320f View commit details
    Browse the repository at this point in the history

  4. Cached timeout time for sim tasks on the sim_modes object.

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    00c29a5 View commit details
    Browse the repository at this point in the history

  5. de-flake tests

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    f6b13b1 View commit details
    Browse the repository at this point in the history

  6. Ensure that mode_settings are deleted from the task proxy when 

    a psuedo-job finishes. Add test for workflow reload.
    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    ff6037c View commit details
    Browse the repository at this point in the history

  7. rationalize tests

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    0cd47a9 View commit details
    Browse the repository at this point in the history

  8. r2r

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    c356bdf View commit details
    Browse the repository at this point in the history

  9. Save correct datetime format to the DB.

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    d6e5aef View commit details
    Browse the repository at this point in the history

  10. Response to review 

    - Re-order the ModeSettings.__init__ method to allow
  db loading before setting the start time.
- Add `try_num` to database.
- Fix/streamline the tests - add explicit broadcast test.
    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    695e5a8 View commit details
    Browse the repository at this point in the history

  11. small test fix

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    a6f7606 View commit details
    Browse the repository at this point in the history

  12. Update cylc/flow/task_job_mgr.py 

    Co-authored-by: Oliver Sanders <[email protected]>
    @wxtim @oliver-sanders
    wxtim and oliver-sanders committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    69b03c9 View commit details
    Browse the repository at this point in the history

  13. Apply suggestions from code review 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    2665d42 View commit details
    Browse the repository at this point in the history

  14. response to review

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    89e5041 View commit details
    Browse the repository at this point in the history

  15. Pass task_events_manager wholesale to sim_time_check. 

    Submit task messages for sim mode outcomes using `task_events_manager` rather than resolvers, to
reduce the risk of ordering problems.
Refactored tests to make them work with the above.
    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    b097d33 View commit details
    Browse the repository at this point in the history

  16. Rationalize existing tests and add a test for a bug discovered by Ronnie

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    7217c00 View commit details
    Browse the repository at this point in the history

  17. Update tests/functional/modes/05-sim-trigger.t

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    970fa1d View commit details
    Browse the repository at this point in the history

  18. add a warning that fail try 1 only does not change on resubmit, only … 

    …retry
    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    312cf28 View commit details
    Browse the repository at this point in the history

  19. Update cylc/flow/simulation.py 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    e2392af View commit details
    Browse the repository at this point in the history

  20. Update cylc/flow/cfgspec/workflow.py 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    08f4345 View commit details
    Browse the repository at this point in the history

  21. fix flake8 issues

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    62d5897 View commit details
    Browse the repository at this point in the history

  22. refactor test broken by sim-mode change

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    3832e7b View commit details
    Browse the repository at this point in the history

  23. Update tests/flakyfunctional/cylc-get-config/04-dummy-mode-output.t 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    8c3bf61 View commit details
    Browse the repository at this point in the history

  24. Apply suggestions from code review 

    Co-authored-by: Oliver Sanders <[email protected]>
    @wxtim @oliver-sanders
    wxtim and oliver-sanders committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    c8d5aea View commit details
    Browse the repository at this point in the history

  25. Fallback created for lack of start time in database. 

    This is likely if someone upgrades a sim mode workflow part-way
through a run.
    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    42aa4b2 View commit details
    Browse the repository at this point in the history

  26. Ensure that broadcasts to fail cycle points triggers a re-parse 

    of fail cycle points.
Check other items in simulation mode settings for similar bugs.
    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    42ad8cc View commit details
    Browse the repository at this point in the history

  27. Use itask summary started time as sole arbiter of simulation 

    tasks which have been submitted before restart.
    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    38db5dc View commit details
    Browse the repository at this point in the history

  28. ensure that broadcast checks and rejects unparsable fail cycle points

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    95d3dc4 View commit details
    Browse the repository at this point in the history

  29. Fix a broken test, and some linting and typing issues

    @wxtim
    wxtim committed Feb 28, 2024
    Configuration menu
    Copy the full SHA
    1058023 View commit details
    Browse the repository at this point in the history

  30. Update cylc/flow/network/resolvers.py

    @wxtim
    wxtim authored Feb 28, 2024
    Configuration menu
    Copy the full SHA
    96db826 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2024

  1. Apply suggestions from code review 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie authored Mar 4, 2024
    Configuration menu
    Copy the full SHA
    0b41424 View commit details
    Browse the repository at this point in the history

  2. Update changes.d/5721.feat.md

    @wxtim
    wxtim authored Mar 4, 2024
    Configuration menu
    Copy the full SHA
    d93de69 View commit details
    Browse the repository at this point in the history

  3. Update cylc/flow/simulation.py

    @wxtim
    wxtim authored Mar 4, 2024
    Configuration menu
    Copy the full SHA
    6268809 View commit details
    Browse the repository at this point in the history

  4. Update changes.d/5721.feat.md 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie authored Mar 4, 2024
    Configuration menu
    Copy the full SHA
    5447530 View commit details
    Browse the repository at this point in the history

  5. fix broken test

    @wxtim
    wxtim committed Mar 4, 2024
    Configuration menu
    Copy the full SHA
    ab53227 View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2024

  1. make sim mode messages _look_ external

    @wxtim
    wxtim committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    d2b372e View commit details
    Browse the repository at this point in the history

  2. removed bc manager changes - one test is failing...

    @wxtim
    wxtim committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    ceb3063 View commit details
    Browse the repository at this point in the history

  3. Prevent totally invalid fail cycle points being accepted for simulation.

    @wxtim
    wxtim committed Mar 5, 2024
    Configuration menu
    Copy the full SHA
    89db69c View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2024

  1. Update cylc/flow/simulation.py 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie authored Mar 6, 2024
    Configuration menu
    Copy the full SHA
    cd2fe55 View commit details
    Browse the repository at this point in the history

  2. ensure than changing fail try 1 only doesn't cause failure

    @wxtim
    wxtim committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    6773246 View commit details
    Browse the repository at this point in the history

  3. Prevent repeated use of sim_task_failed giving different answers (bas… 

    …ed on the previous answer).
    @wxtim
    wxtim committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    d147b0f View commit details
    Browse the repository at this point in the history

  4. fix

    @wxtim
    wxtim committed Mar 6, 2024
    Configuration menu
    Copy the full SHA
    269078a View commit details
    Browse the repository at this point in the history

Commits on Mar 11, 2024

  1. Broadcast changes to simulated tasks in task_job_manager 

    to allow non simulation settings to be modified and
to avoid changing the itask.tdef.rtconfig.
    @wxtim
    wxtim committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    b4f4bb1 View commit details
    Browse the repository at this point in the history

  2. test that clearing broadcasts works for sim tasks

    @wxtim
    wxtim committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    370d2b6 View commit details
    Browse the repository at this point in the history

  3. Update cylc/flow/simulation.py 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie authored Mar 11, 2024
    Configuration menu
    Copy the full SHA
    b9e1d15 View commit details
    Browse the repository at this point in the history

  4. Update cylc/flow/simulation.py 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie authored Mar 11, 2024
    Configuration menu
    Copy the full SHA
    b355ae2 View commit details
    Browse the repository at this point in the history

  5. Update tests/integration/test_simulation.py 

    Co-authored-by: Ronnie Dutta <[email protected]>
    @wxtim @MetRonnie
    wxtim and MetRonnie authored Mar 11, 2024
    Configuration menu
    Copy the full SHA
    44abd5a View commit details
    Browse the repository at this point in the history

  6. fix test

    @wxtim
    wxtim committed Mar 11, 2024
    Configuration menu
    Copy the full SHA
    cb064a0 View commit details
    Browse the repository at this point in the history

Commits on Mar 14, 2024

  1. Merge branch 'master' into feature.sim_mode_at_runtime

    @hjoliver
    hjoliver authored Mar 14, 2024
    Configuration menu
    Copy the full SHA
    ef2fc58 View commit details
    Browse the repository at this point in the history

  2. Update changes.d/5721.feat.md 

    Co-authored-by: Hilary James Oliver <[email protected]>
    @wxtim @hjoliver
    wxtim and hjoliver authored Mar 14, 2024
    Configuration menu
    Copy the full SHA
    6be7372 View commit details
    Browse the repository at this point in the history

  3. Update cylc/flow/simulation.py 

    Co-authored-by: Hilary James Oliver <[email protected]>
    @wxtim @hjoliver
    wxtim and hjoliver authored Mar 14, 2024
    Configuration menu
    Copy the full SHA
    389a0be View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2024

  1. Fix func test, after update on master.

    @hjoliver
    hjoliver committed Mar 15, 2024
    Configuration menu
    Copy the full SHA
    9186ad1 View commit details
    Browse the repository at this point in the history