Fix.simulation mode started bug

[wxtim]

Fixes a bug discovered by @oliver-sanders in the process of reviewing skip mode work (#6039 (comment)). The bug however, is long-standing (it goes back at least as far as Cylc 8.0.0.

Summary

In simulation mode, job starting is done by the job submission pathway as a side-effect (because in sim mode the two might as well be identical). However, the pathway is not currently spawning children of started.

Example

Run this

# flow.cylc
[scheduling]
    [[graph]]
        R1 = a? & a:started => b

[runtime]
  [[root]]
    run mode  = skip

Check List

• I have read CONTRIBUTING.md and added my name as a Code Contributor.
• Contains logically grouped changes (else tidy your branch by rebase).
• Does not contain off-topic changes (use other PRs for other changes).
• Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
• Tests are included (or explain why tests are not needed).
• Changelog entry included if this is a change that can affect users
• Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
• If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

[oliver-sanders]

LGTM.

This is a bug (note just a shortcoming in simulation mode) because (as Tim noted) the DB lists the "started" output, but the downstream task is never triggered which is inconsistent.

I did wonder whether this should be fixed by getting the submit code to send the started output like so:

diff --git a/cylc/flow/task_job_mgr.py b/cylc/flow/task_job_mgr.py
index 185966ff1..7a6451605 100644
--- a/cylc/flow/task_job_mgr.py
+++ b/cylc/flow/task_job_mgr.py
@@ -1028,6 +1028,9 @@ class TaskJobManager:
             self.task_events_mgr.process_message(
                 itask, INFO, TASK_OUTPUT_SUBMITTED,
             )
+            self.task_events_mgr.process_message(
+                itask, INFO, TASK_OUTPUT_STARTED,
+            )
             self.workflow_db_mgr.put_insert_task_jobs(
                 itask, {
                     'time_submit': now_str,

But quickly realised why we don't do that:

ERROR - 'execution polling intervals'
    Traceback (most recent call last):
    ...
    KeyError: 'execution polling intervals'

This makes Cylc think that there is actually a job running, it tries to setup health checks which fail for obvious reasons.

[hjoliver]

LGTM. Not merging in case @oliver-sanders comment requires a small change?

[wxtim]

@oliver-sanders - final check that you are happy with my response to your comment?

[oliver-sanders]

Not sure which one, but yes I'm happy!