@@ -17,365 +17,383 @@
| python SDK | +yaml | +
|---|---|
-## Example Run +Example present at: ```examples/python-tasks.py``` - +Run it as: ```python examples/python-tasks.py``` -To give you a flavour of how magnus works, lets create a simple pipeline. +```python +from magnus import Pipeline, Task -Copy the contents of this yaml into getting-started.yaml or alternatively in a python file if you are using the SDK. +def main(): + step1 = Task( + name="step1", + command="examples.functions.return_parameter", + ) + step2 = Task( + name="step2", + command="examples.functions.display_parameter", + terminate_with_success=True, + ) ---- -!!! Note + step1 >> step2 - The below execution would create a folder called 'data' in the current working directory. - The command as given should work in linux/macOS but for windows, please change accordingly. + pipeline = Pipeline( + start_at=step1, + steps=[step1, step2], + add_terminal_nodes=True, + ) ---- + pipeline.execute() + + +if __name__ == "__main__": + main() +``` - - + |
-``` yaml
++ +Example present at: ```examples/python-tasks.yaml``` + + +Execute via the cli: ```magnus execute -f examples/python-tasks.yaml``` + +```yaml dag: - description: Getting started - start_at: step parameters + description: | + This is a simple pipeline that does 3 steps in sequence. + In this example: + 1. First step: returns a "parameter" x as a Pydantic model + 2. Second step: Consumes that parameter and prints it + + This pipeline demonstrates one way to pass small data from one step to another. + + start_at: step 1 steps: - step parameters: + step 1: type: task - command_type: python-lambda - command: "lambda x: {'x': int(x) + 1}" - next: step shell - step shell: + command_type: python # (2) + command: examples.functions.return_parameter # (1) + next: step 2 + step 2: type: task - command_type: shell - command: mkdir data ; env >> data/data.txt # For Linux/macOS + command_type: python + command: examples.functions.display_parameter next: success - catalog: - put: - - "*" success: type: success fail: type: fail ``` -The same could also be defined via a Python SDK. - -```python - -#in pipeline.py -from magnus import Pipeline, Task + |
-def pipeline():
- first = Task(name='step parameters', command="lambda x: {'x': int(x) + 1}", command_type='python-lambda',
- next_node='step shell')
- second = Task(name='step shell', command='mkdir data ; env >> data/data.txt',
- command_type='shell', catalog={'put': '*'})
+