Merge pull request #151 from AstraZeneca/improved_docs

docs: working on it

docs/concepts/catalog.md

@@ -1,5 +1,7 @@
 [tasks](task.md) might also need to pass ```files``` between them.
 
+## Concept
+
 For example:
 
 ```python linenums="1"
@@ -20,7 +22,7 @@ consume()
 ```
 
 
-## Runnable representation
+## Syntax
 
 The same can be represented in ```runnable``` as [catalog](../reference.md/#catalog).
 

docs/concepts/index.md

@@ -41,7 +41,7 @@ pipeline.execute()
 ```
 
 
-- ```runnable``` exposes the functions ```generate``` and ```consume``` as [tasks](task.md).
+- ```runnable``` wraps the functions ```generate``` and ```consume``` as [tasks](task.md).
 - Tasks can [access and return](parameters.md/#access_returns) parameters.
 - Tasks can also share files between them using [catalog](catalog.md).
 - Tasks are stitched together as [pipeline](pipeline.md)

docs/concepts/parameters.md

@@ -1,5 +1,7 @@
 ```parameters``` are data that can be passed from one ```task``` to another.
 
+## Concept
+
 For example, in the below snippet, the parameters ```x``` and ```y``` are passed from
 ```generate``` to ```consume```.
 

docs/concepts/pipeline.md

@@ -15,6 +15,7 @@ A ```workflow``` is a sequence of ```steps``` to perform.
 
 <br>
 
+## Concept
 
 A visual example of a workflow:
 
@@ -57,20 +58,28 @@ and [stub](task.md/#stub).
 
 === "sdk"
 
+    - [x] The first step of the ```steps``` is the start of the workflow.
+    - [x] The order of execution follows the order of the tasks in the list.
+    - [x] The terminal nodes ```success``` and ```fail``` are added automatically.
+
     ```python linenums="1"
     --8<-- "examples/02-sequential/traversal.py"
     ```
 
     1. Start the pipeline.
     2. The order of the steps is the execution order
 
-    - [x] The first step of the ```steps``` is the start of the workflow.
-    - [x] The order of execution follows the order of the tasks in the list.
-    - [x] The terminal nodes ```success``` and ```fail``` are added automatically.
+
+
 
 
 === "yaml"
 
+    - [x] The first step  is the step corresponding to ```start_at```
+    - [x] The mapping defined in the steps.
+    - [x] The ```next``` step after a successful execution of a ```step```.
+    - [x] ```success``` as ```next``` node implies successful execution of the pipeline.
+
     ```yaml linenums="1"
     --8<-- "examples/02-sequential/traversal.yaml"
     ```
@@ -80,10 +89,7 @@ and [stub](task.md/#stub).
     3. Add the success and fail nodes.
 
 
-    - [x] The first step  is the step corresponding to ```start_at```
-    - [x] The mapping defined in the steps.
-    - [x] The ```next``` step after a successful execution of a ```step```.
-    - [x] Needs explicit definition of ```success``` and ```fail``` nodes.
+
 
 <br>
 
@@ -93,8 +99,7 @@ and [stub](task.md/#stub).
 By default, any failure during the execution of step will traverse to ```fail``` node
 marking the execution as failed.
 
-The ```fail``` node is implicitly added to the pipeline in python SDK while it
-has to be stated in the yaml.
+The ```fail``` node is implicitly added to the pipeline.
 
 
 This behavior can be over-ridden to follow a different path based on expected failures.

docs/concepts/task.md

@@ -12,6 +12,8 @@ the python function/notebook/shell script/stubs.
 
 ## Python functions
 
+Uses python functions as tasks.
+
 [API Documentation](../reference.md/#pythontask)
 
 ### Example
@@ -106,7 +108,7 @@ ecosystem while shell provides a interface to non-python executables.
 
 === "yaml"
 
-    ```yaml linenums="1" hl_lines="19-23"
+    ```yaml linenums="1" hl_lines="16-23"
     --8<-- "examples/01-tasks/scripts.yaml"
     ```
 

docs/reference.md

@@ -1,44 +1,88 @@
-## Catalog
+## PythonTask
 
 === "sdk"
 
-    ::: runnable.Catalog
+    ::: runnable.PythonTask
         options:
             show_root_heading: true
             show_bases: false
+            show_docstring_description: true
             heading_level: 3
 
 === "yaml"
 
-
+    Attributes:
+
+    - ```name```: the name of the task
+    - ```command```: the dotted path reference to the function.
+    - ```next```: the next node to call if the function succeeds. Use ```success``` to terminate
+    the pipeline successfully or ```fail``` to terminate with fail.
+    - ```on_failure```: The next node in case of failure.
+    - ```catalog```: mapping of cataloging items
+    - ```overrides```: mapping of step overrides from global configuration.
+
+    ```yaml
+    dag:
+      steps:
+        name: <>
+          type: task
+          command: <>
+          next: <>
+          on_failure: <>
+          catalog: # Any cataloging to be done.
+          overrides: # mapping of overrides of global configuration
+    ```
 
 <hr style="border:2px dotted orange">
 
-## Stub
+
+## NotebookTask
 
 === "sdk"
 
-    ::: runnable.Stub
+    ::: runnable.NotebookTask
         options:
             show_root_heading: true
             show_bases: false
+            show_docstring_description: true
             heading_level: 3
 
 === "yaml"
 
+    Attributes:
+
+    - ```name```: the name of the task
+    - ```command```: the path to the notebook relative to the project root.
+    - ```next```: the next node to call if the function succeeds. Use ```success``` to terminate
+    the pipeline successfully or ```fail``` to terminate with fail.
+    - ```on_failure```: The next node in case of failure.
+    - ```catalog```: mapping of cataloging items
+    - ```overrides```: mapping of step overrides from global configuration.
+
+    ```yaml
+    dag:
+      steps:
+        name: <>
+          type: task
+          command: <>
+          next: <>
+          on_failure: <>
+          catalog: # Any cataloging to be done.
+          overrides: # mapping of overrides of global configuration
+    ```
 
 
 <hr style="border:2px dotted orange">
 
-## PythonTask
+
+## Catalog
 
 === "sdk"
 
-    ::: runnable.PythonTask
+    ::: runnable.Catalog
         options:
             show_root_heading: true
             show_bases: false
-            show_docstring_description: true
             heading_level: 3
 
 === "yaml"
@@ -47,15 +91,14 @@
 
 <hr style="border:2px dotted orange">
 
-## ShellTask
+## Stub
 
 === "sdk"
 
-    ::: runnable.ShellTask
+    ::: runnable.Stub
         options:
             show_root_heading: true
             show_bases: false
-            show_docstring_description: true
             heading_level: 3
 
 === "yaml"
@@ -65,11 +108,12 @@
 <hr style="border:2px dotted orange">
 
 
-## NotebookTask
+
+## ShellTask
 
 === "sdk"
 
-    ::: runnable.NotebookTask
+    ::: runnable.ShellTask
         options:
             show_root_heading: true
             show_bases: false
@@ -79,8 +123,12 @@
 === "yaml"
 
 
+
 <hr style="border:2px dotted orange">
 
+
+
+
 ## Parallel
 
 

examples/06-parallel/parallel.py

@@ -15,7 +15,7 @@
 from runnable import NotebookTask, Parallel, Pipeline, PythonTask, ShellTask, Stub
 
 
-def traversal(execute: bool = True):
+def traversal():
     """
     Use the pattern of using "execute" to control the execution of the pipeline.
 
@@ -46,17 +46,14 @@ def traversal(execute: bool = True):
     # The order of execution follows the order of the tasks in the list.
     pipeline = Pipeline(steps=[stub_task, python_task, shell_task, notebook_task])
 
-    if execute:  # Do not execute the pipeline if we are using it as a branch
-        pipeline.execute()
-
     return pipeline
 
 
 def main():
     parallel_step = Parallel(
         name="parallel_step",
         terminate_with_success=True,
-        branches={"branch1": traversal(execute=False), "branch2": traversal(execute=False)},
+        branches={"branch1": traversal(), "branch2": traversal()},
     )
 
     pipeline = Pipeline(steps=[parallel_step])