Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

reload: reloading shakes out runahead tasks #5825

Closed
oliver-sanders opened this issue Nov 17, 2023 · 2 comments
Closed

reload: reloading shakes out runahead tasks #5825

oliver-sanders opened this issue Nov 17, 2023 · 2 comments
Assignees
@hjoliver
Labels
bug
Milestone
cylc-8.2.4

Comments

@oliver-sanders
Copy link
Member

oliver-sanders commented Nov 17, 2023 •
edited
Loading

There's a nasty interaction between reload and runahead.

[scheduler]                    
    allow implicit tasks = True    
                               
[scheduling]                   
    initial cycle point = 2000
    runahead limit = P5D       
    [[graph]]                  
        P1D = """              
            a => b             
        """                    
                               
[runtime]                      
    [[root]]                   
        script = sleep 500

When this workflow starts, the tasks 2000 through 2005 are submitted:

2023-11-17T10:05:39Z INFO - [2000/a submitted job:01 flows:1] => running
2023-11-17T10:05:39Z INFO - [2000/a running job:01 flows:1] health: execution timeout=None, polling intervals=PT1H,...
2023-11-17T10:05:39Z INFO - [2002/a submitted job:01 flows:1] => running
2023-11-17T10:05:39Z INFO - [2002/a running job:01 flows:1] health: execution timeout=None, polling intervals=PT1H,...
2023-11-17T10:05:39Z INFO - [2003/a submitted job:01 flows:1] => running
2023-11-17T10:05:39Z INFO - [2003/a running job:01 flows:1] health: execution timeout=None, polling intervals=PT1H,...
2023-11-17T10:05:39Z INFO - [2004/a submitted job:01 flows:1] => running
2023-11-17T10:05:39Z INFO - [2004/a running job:01 flows:1] health: execution timeout=None, polling intervals=PT1H,...
2023-11-17T10:05:39Z INFO - [2005/a submitted job:01 flows:1] => running
2023-11-17T10:05:39Z INFO - [2005/a running job:01 flows:1] health: execution timeout=None, polling intervals=PT1H,...

But if I reload it, the task 2006 submits:

2023-11-17T10:05:40Z INFO - Reloading task definitions.
2023-11-17T10:05:40Z INFO - [2000/a running job:01 flows:1] reloaded task definition
2023-11-17T10:05:40Z WARNING - [2000/a running job:01 flows:1] active with pre-reload settings
2023-11-17T10:05:40Z INFO - [2001/a running job:01 flows:1] reloaded task definition
2023-11-17T10:05:40Z WARNING - [2001/a running job:01 flows:1] active with pre-reload settings
2023-11-17T10:05:40Z INFO - [2002/a running job:01 flows:1] reloaded task definition
2023-11-17T10:05:40Z WARNING - [2002/a running job:01 flows:1] active with pre-reload settings
2023-11-17T10:05:40Z INFO - [2003/a running job:01 flows:1] reloaded task definition
2023-11-17T10:05:40Z WARNING - [2003/a running job:01 flows:1] active with pre-reload settings
2023-11-17T10:05:40Z INFO - [2004/a running job:01 flows:1] reloaded task definition
2023-11-17T10:05:40Z WARNING - [2004/a running job:01 flows:1] active with pre-reload settings
2023-11-17T10:05:40Z INFO - [2005/a running job:01 flows:1] reloaded task definition
2023-11-17T10:05:40Z WARNING - [2005/a running job:01 flows:1] active with pre-reload settings
2023-11-17T10:05:40Z INFO - [2006/a waiting(runahead) job:00 flows:1] reloaded task definition
2023-11-17T10:05:40Z INFO - LOADING job data
2023-11-17T10:05:40Z INFO - [2006/a waiting(runahead) job:00 flows:1] => waiting
2023-11-17T10:05:40Z INFO - [2006/a waiting job:00 flows:1] => waiting(queued)

Reload again and 2007 submits:

2023-11-17T10:05:45Z INFO - Reloading task definitions.
...
2023-11-17T10:05:45Z INFO - [2007/a waiting(runahead) job:00 flows:1] => waiting
2023-11-17T10:05:45Z INFO - [2007/a waiting job:00 flows:1] => waiting(queued)
2023-11-17T10:05:45Z INFO - Reload completed.

Reload again and 2008 submits:

2023-11-17T10:05:48Z INFO - Reloading task definitions.
...
2023-11-17T10:05:49Z INFO - [2008/a waiting(runahead) job:00 flows:1] => waiting
2023-11-17T10:05:49Z INFO - [2008/a waiting job:00 flows:1] => waiting(queued)

And so on.
@oliver-sanders oliver-sanders added this to the cylc-8.2.4 milestone Nov 17, 2023
@wxtim wxtim mentioned this issue Nov 20, 2023
Merged
8 tasks
@oliver-sanders oliver-sanders self-assigned this Nov 21, 2023
@oliver-sanders
Copy link
Member Author

The issue seems to be caused by compute_runahead counting cycles with runahead tasks as active.

oliver-sanders added a commit to oliver-sanders/cylc-flow that referenced this issue Nov 21, 2023
@oliver-sanders
task_pool: fix compute_runahead calculation
40db21c 
* Closes cylc#5825
* Cycles were considered active if they contained runahead limited
  tasks.
* This could cause the runahead limit to be bumped forwards whenever the
  limit calculation was forced to update, e.g. on reload.
* This filters out tasks at or beyond the runahead limit and straigntens
  out the task status checks to match Cylc 7 behaviour in compat mode.
@oliver-sanders oliver-sanders mentioned this issue Nov 21, 2023
Closed
8 tasks
oliver-sanders added a commit to oliver-sanders/cylc-flow that referenced this issue Nov 22, 2023
@oliver-sanders
task_pool: fix compute_runahead calculation
8d76a96 
* Closes cylc#5825
* Cycles were considered active if they contained runahead limited
  tasks.
* This could cause the runahead limit to be bumped forwards whenever the
  limit calculation was forced to update, e.g. on reload.
* This filters out tasks at or beyond the runahead limit and straigntens
  out the task status checks to match Cylc 7 behaviour in compat mode.
oliver-sanders added a commit to oliver-sanders/cylc-flow that referenced this issue Nov 28, 2023
@oliver-sanders
task_pool: fix compute_runahead calculation
64f35f4 
* Closes cylc#5825
* Cycles were considered active if they contained runahead limited
  tasks.
* This could cause the runahead limit to be bumped forwards whenever the
  limit calculation was forced to update, e.g. on reload.
* This filters out tasks at or beyond the runahead limit and straigntens
  out the task status checks to match Cylc 7 behaviour in compat mode.
@oliver-sanders oliver-sanders mentioned this issue Dec 20, 2023
Merged
8 tasks
oliver-sanders added a commit to oliver-sanders/cylc-flow that referenced this issue Dec 20, 2023
@oliver-sanders
reload: test reloading doesn't nudge the runahead limit
be08c7e 
* Closes cylc#5825
@oliver-sanders oliver-sanders mentioned this issue Dec 20, 2023
Merged
hjoliver pushed a commit to hjoliver/cylc-flow that referenced this issue Jan 8, 2024
@oliver-sanders @hjoliver
reload: test reloading doesn't nudge the runahead limit
0ca9dda 
* Closes cylc#5825
@hjoliver
Copy link
Member

hjoliver commented Jan 9, 2024

Closed by #5893

@hjoliver hjoliver closed this as completed Jan 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hjoliver hjoliver

Labels
bug
Projects
None yet
Milestone
cylc-8.2.4
Development

Successfully merging a pull request may close this issue.

task_pool: fix compute_runahead calculation oliver-sanders/cylc-flow
2 participants
@hjoliver @oliver-sanders