Optional output extension

src/7-to-8/major-changes/cylc-install.rst

@@ -118,21 +118,30 @@ See :ref:`the user guide <RemoteInit>` for more details.
 Migrating From ``rose suite-run``
 ---------------------------------
 
-The ``rose suite-run`` command has been replaced by ``cylc install``.
+The ``rose suite-run`` command did three things:
 
-.. code-block:: bash
+1. It validated the workflow.
+2. It installed the workflow into the ``cylc-run`` directory.
+3. It started a scheduler to run the workflow.
 
-   # rose 2019 / Cylc 7
-   $ rose suite-run
+At Cylc 8, there is a separate command for each of these three things:
 
-   # rose 2 / Cylc 8
+.. code-block:: console
+
+   $ cylc validate
    $ cylc install
-   $ cylc play <id>
+   $ cylc play
+
+But for convenience there is a compound command which does all three:
+
+.. code-block:: console
+
+   $ cylc vip  # vip=validate-install-play
 
-Support for the ``rose-suite.conf`` file is provided by the :ref:`Cylc Rose`
+Support for the ``rose-suite.conf`` file is now provided by the :ref:`Cylc Rose`
 plugin which must be installed for Rose integration.
 
-.. spoiler:: Installation
+.. spoiler:: Installation Details
    :class: hint
 
    See the :ref:`installation` section for instructions.

src/7-to-8/major-changes/suicide-triggers.rst

@@ -91,6 +91,9 @@ Also, the suicide triggers can be removed.
    - # Remove the "foo" task in the fail case.
    - recover => ! foo
 
+At least one optional output must be generated, so in this example ``foo`` must
+either succeed or fail.
+
 In Cylc 7, suicide triggers were used to remove tasks that did not complete
 during runtime. Cylc 8's event-driven graph handling allows such graph
 branching using optional output syntax, without the need for suicide triggers.

src/7-to-8/summary.rst

@@ -39,9 +39,11 @@ Upgrading To Cylc 8
 
 There have been some configuration changes at Cylc 8.
 To upgrade your Cylc 7 suite to a Cylc 8 workflow, first make sure it validates
-in Cylc 7 without any warnings, then rename the workflow configuration file
-from ``suite.rc`` to  ``flow.cylc``, then run ``cylc validate`` in Cylc 8 and
-take action on any warnings.
+in Cylc 7 without any warnings, you can now run it in compatibility mode.
+
+To upgrade the workflow to run without compatibility mode, rename the workflow
+configuration file from ``suite.rc`` to  ``flow.cylc``, run ``cylc validate``
+in Cylc 8 and take action on any warnings.
 
 .. TODO Add ref to breaking changes section within Major changes, once created,
    including optional ouputs.
@@ -71,21 +73,23 @@ At Cylc 8, there are two UIs available to monitor and control your workflows:
 
    .. code-block:: bash
 
-      cylc gui
+      cylc gui <workflow_id>
 
 Command Changes
 ---------------
 
-``cylc run <suite_name>`` at Cylc 7 has become ``cylc play <workflow_id>``.
+``cylc run <suite_name>`` has become ``cylc play <workflow_id>``.
+
+``rose suite-run`` is now ``cylc vip`` (**V**\ alidate, **I**\ nstall, **P**\ lay).
 
 .. seealso::
 
    * User Guide: :ref:`WorkflowStartUp`
    * Major Changes: :ref:`728.play_pause_stop`
    * Major Changes: :ref:`MajorChangesCLI`
 
-At Cylc 8, use ``cylc pause <workflow_id>`` to pause a workflow, halting all job
-submission. To restart the workflow, use ``cylc play <workflow_id>``.
+At Cylc 8, use ``cylc pause <workflow_id>`` to pause a workflow, preventing new
+job submissions. To restart the workflow, use ``cylc play <workflow_id>``.
 
 To start a fresh run, use ``cylc install`` and play it safely in the new run
 directory.
@@ -96,12 +100,13 @@ Task/Job States
 ---------------
 
 :term:`Tasks <task>` are nodes in the abstract workflow graph, representing
-applications to run at the appropriate point in the workflow. A :term:`job <job>`
+applications to run at the appropriate point in the workflow.
+
+A :term:`job <job>`
 is the script (and subsequent process) submitted by Cylc to
 actually run the application. A task can have multiple jobs as the result of
 automatic retries or manual re-triggering.
 
-
 The 13 task/job states in Cylc 7 have been simplified to 8. Tasks and jobs have been
 separated and states of both can be viewed in the GUI.
 

src/glossary.rst

@@ -1479,8 +1479,12 @@ Glossary
    optional output
       Optional :term:`task outputs <task output>` are marked with a question
       mark in the :term:`graph`, e.g. ``foo:x?``, or ``foo:fail?``,  or
-      ``foo?`` (short for ``foo:succeed?``). The may or may not be completed at
-      runtime. Optional outputs are primarily used for :term:`graph branching`.
+      ``foo?`` (short for ``foo:succeed?``).
+
+      One or more optional outputs must be generated when the task runs,
+      otherwise the workflow will :term:`stall`. This allows the workflow
+      to follow different pathways through the graph according to runtime
+      events, this is called :term:`graph branching`.
 
       .. seealso::
 
@@ -1489,7 +1493,6 @@ Glossary
 
 
    required output
-   expected output
       Task outputs that are not marked as :term:`optional <optional output>`
       in the :term:`graph` are required to be completed at runtime. If not, the
       :term:`scheduler` retains the task as :term:`incomplete` pending user

src/reference/changes.rst

@@ -20,6 +20,112 @@ For more detail see the component changelogs:
 * `metomi-rose-changelog`_
 * `metomi-isodatetime-changelog`_
 
+----------
+
+Cylc 8.3.0
+----------
+
+.. TODO - uncomment before 8.3.0 release
+
+   .. admonition:: Cylc Components
+      :class: hint
+
+      :cylc-flow: `8.3 <https://github.com/cylc/cylc-flow/blob/8.3.x/CHANGES.md>`__
+      :cylc-uiserver: `1.4 <https://github.com/cylc/cylc-uiserver/blob/1.4.x/CHANGES.md>`__
+      :cylc-rose: `1.4 <https://github.com/cylc/cylc-rose/blob/1.4.x/CHANGES.md>`__
+
+Optional Outputs
+^^^^^^^^^^^^^^^^
+
+Optional outputs (introduced in Cylc 8.0.0) are a way of validating the outcome
+of a task to ensure it "completed" as intended. E.G, by default, if a task
+fails, it will considered :term:`incomplete <incomplete task>` and the
+workflow will stall.
+
+Before (Cylc 8.0.0 - 8.1.4)
+   All optional outputs were completely "optional",
+   that is to say, they may or may not be generated when a task runs.
+
+From Cylc 8.3.0 onwards
+   When optional outputs are defined in a workflow, Cylc
+   now requires one or more of them to be generated by the task at runtime.
+
+So for this example:
+
+.. code-block:: cylc-graph
+
+   a:x => x
+   a:y => y
+   x | y => b
+
+The task must succeed (success is required unless marked otherwise) and at
+least one of the outputs ``x`` or ``y`` must be generated.
+
+In Python syntax that condition looks like this:
+
+.. code-block:: python
+
+   succeeded and (x or y)
+
+If this condition is not met, then the task will be considered
+:term:`incomplete <incomplete task>` altering you to the problem.
+
+This change solves an issue where the pathway through the graph could be
+interrupted.
+E.G. in the above example, the intention is that Cylc will follow either the
+``a:x`` branch *or* the ``a:y`` branch, however, with Cylc <=8.1.4 it is
+possible that neither the ``x`` or ``y`` outputs would be generated, in which
+case the task ``b`` would never be run.
+
+This behaviour can be overridden using the
+:cylc:conf:`[runtime][<namespace>]completion` configuration.
+E.G. To restore the old behaviour
+
+.. code-block:: cylc
+
+   [runtime]
+       [[a]]
+           # this sets the condition that Cylc uses to determine whether a
+           # task has completed
+           completion = succeeded or failed
+
+The new ``completion`` configuration opens up the potential for much
+more powerful task diagnostics with :term:`custom outputs <custom output>`.
+
+See :ref:`Graph Branching` for more information about optional outputs
+and examples including the new ``completion`` configuration.
+
+
+Task Expiry
+^^^^^^^^^^^
+
+As a result of the optional outputs change above, task expiry can now be used
+in combination with optional outputs e.g:
+
+.. code-block:: cylc-graph
+
+   # run "x" if "a" succeeds
+   a? => x
+   # run "y" if "a" expires
+   a:expired? => y
+
+In this example, ``a:fail?`` is not handled in the graph, so if ``a`` fails,
+it will be incomplete and the workflow will stall altering you to the problem.
+
+This change means you need to add a ``?`` mark symbol after ``:expired``
+triggers in the graph.
+
+The new ``completion`` configuration can also be used in combination with
+task expiry e.g:
+
+.. code-block:: cylc
+
+   [runtime]
+       [[a]]
+           # this task is allowed to succeed or expire but not fail
+           completion = succeeded or expired
+
+
 ----------
 
 Cylc 8.2.0
@@ -28,9 +134,9 @@ Cylc 8.2.0
 .. admonition:: Cylc Components
    :class: hint
 
-   :cylc-flow: `8.2 <https://github.com/cylc/cylc-flow/blob/master/CHANGES.md>`__
-   :cylc-uiserver: `1.3 <https://github.com/cylc/cylc-uiserver/blob/master/CHANGES.md>`__
-   :cylc-rose: `1.3 <https://github.com/cylc/cylc-rose/blob/master/CHANGES.md>`__
+   :cylc-flow: `8.2 <https://github.com/cylc/cylc-flow/blob/8.2.x/CHANGES.md>`__
+   :cylc-uiserver: `1.3 <https://github.com/cylc/cylc-uiserver/blob/1.3.x/CHANGES.md>`__
+   :cylc-rose: `1.3 <https://github.com/cylc/cylc-rose/blob/1.3.x/CHANGES.md>`__
 
 
 Configure The Default View
@@ -126,11 +232,11 @@ Combined Commands
 
 Two new commands have been added as short-cuts for common working patterns:
 
-``cylc vip`` 
+``cylc vip``
    Validate, install and plays a workflow, equivalent to:
 
    .. code-block:: bash
-   
+
       cylc validate <path>
       cylc install <path>
       cylc play <id>