Make packaged kedro project return session.run() output (#4139)

* Made run command return output Signed-off-by: Elena Khaustova <[email protected]> * Updated docs Signed-off-by: Elena Khaustova <[email protected]> * Updated release notes Signed-off-by: Elena Khaustova <[email protected]> * Updated empty project template Signed-off-by: Elena Khaustova <[email protected]> * Update RELEASE.md Apply suggested change Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: ElenaKhaustova <[email protected]> * Update docs/source/tutorial/package_a_project.md Apply suggested change Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: ElenaKhaustova <[email protected]> * Update docs/source/tutorial/package_a_project.md Apply suggested change Co-authored-by: Merel Theisen <[email protected]> Signed-off-by: ElenaKhaustova <[email protected]> * Updated release notes Signed-off-by: Elena Khaustova <[email protected]> * Updated outdated prefer docs refs Signed-off-by: Elena Khaustova <[email protected]> * Updated outdated prefer docs refs Signed-off-by: Elena Khaustova <[email protected]> * Updated outdated prefer docs refs Signed-off-by: Elena Khaustova <[email protected]> --------- Signed-off-by: Elena Khaustova <[email protected]> Signed-off-by: ElenaKhaustova <[email protected]> Co-authored-by: Merel Theisen <[email protected]>
kedro-org · Sep 5, 2024 · 7633267 · 7633267
1 parent a10690a
commit 7633267
Show file tree

Hide file tree

Showing 5 changed files with 24 additions and 9 deletions.
diff --git a/RELEASE.md b/RELEASE.md
@@ -1,6 +1,7 @@
 # Upcoming Release
 
 ## Major features and improvements
+* Made packaged Kedro projects return `session.run()` output to be used when running it in the interactive environment.
 * Enhanced `OmegaConfigLoader` configuration validation to detect duplicate keys at all parameter levels, ensuring comprehensive nested key checking.
 ## Bug fixes and other changes
 * Fixed bug where using dataset factories breaks with `ThreadRunner`.

diff --git a/docs/source/deployment/prefect.md b/docs/source/deployment/prefect.md
@@ -2,7 +2,7 @@
 
 This page explains how to run your Kedro pipeline using [Prefect 2.0](https://www.prefect.io/opensource), an open-source workflow management system.
 
-The scope of this documentation is the deployment to a self hosted [Prefect Server](https://docs.prefect.io/2.10.17/host/), which is an open-source backend that makes it easy to monitor and execute your Prefect flows and automatically extends Prefect 2.0. We will use an [Agent that dequeues submitted flow runs from a Work Queue](https://docs.prefect.io/2.10.17/tutorial/deployments/#why-workpools-and-workers).
+The scope of this documentation is the deployment to a self hosted [Prefect Server](https://docs-2.prefect.io/latest/guides/host/), which is an open-source backend that makes it easy to monitor and execute your Prefect flows and automatically extends Prefect 2.0. We will use an [Agent that dequeues submitted flow runs from a Work Queue](https://docs-2.prefect.io/latest/concepts/deployments/#workers-and-work-pools).
 
 ```{note}
 This deployment has been tested using Kedro 0.18.10 with Prefect version 2.10.17. If you want to deploy with Prefect 1.0, we recommend you review [earlier versions of Kedro's Prefect deployment documentation](https://docs.kedro.org/en/0.18.9/deployment/prefect.html).
@@ -12,7 +12,7 @@ This deployment has been tested using Kedro 0.18.10 with Prefect version 2.10.17
 
 To use Prefect 2.0 and Prefect Server, ensure you have the following prerequisites in place:
 
-- [Prefect 2.0 is installed](https://docs.prefect.io/2.10.17/getting-started/installation/#installing-the-latest-version) on your machine
+- [Prefect 2.0 is installed](https://docs-2.prefect.io/latest/getting-started/installation/#installing-the-prefect-2-latest-version) on your machine
 
 ## Setup
 
@@ -40,7 +40,7 @@ Run a Prefect Server instance:
 prefect server start
 ```
 
-In a separate terminal, [create a work pool](https://docs.prefect.io/2.10.17/concepts/work-pools/#work-pool-configuration) to organise the work and [create a work queue](https://docs.prefect.io/2.10.17/concepts/work-pools/#work-queues) for your agent to pull from:
+In a separate terminal, [create a work pool](https://docs-2.prefect.io/latest/concepts/work-pools/#work-pool-configuration) to organise the work and [create a work queue](https://docs-2.prefect.io/latest/concepts/work-pools/#work-queues) for your agent to pull from:
 
 ```bash
 prefect work-pool create --type prefect-agent <work_pool_name>
@@ -57,7 +57,7 @@ prefect agent start --pool <work_pool_name> --work-queue <work_queue_name>
 
 ### Convert your Kedro pipeline to Prefect 2.0 flow
 
-To build a [Prefect flow](https://docs.prefect.io/core/concepts/flows.html) for your Kedro pipeline programmatically and register it with the Prefect API, use the following Python script, which should be stored in your project’s **root directory**:
+To build a [Prefect flow](https://docs-2.prefect.io/latest/concepts/flows/) for your Kedro pipeline programmatically and register it with the Prefect API, use the following Python script, which should be stored in your project’s **root directory**:
 
 ```python
 # <project_root>/register_prefect_flow.py
@@ -250,7 +250,7 @@ Be sure that your Prefect Server is up and running. Verify that the deployment s
 
 ### Run Prefect flow
 
-Now, having the flow registered, you can use [Prefect Server UI](https://docs.prefect.io/2.10.17/host/) to orchestrate and monitor it.
+Now, having the flow registered, you can use [Prefect Server UI](https://docs-2.prefect.io/latest/guides/host/) to orchestrate and monitor it.
 
 Navigate to http://localhost:4200/deployments to see your registered flow.
 

diff --git a/docs/source/tutorial/package_a_project.md b/docs/source/tutorial/package_a_project.md
@@ -144,6 +144,19 @@ main(
 
 This is equivalent to `python -m <package_name>` at the command line, and you can pass in all the arguments that correspond to the options described by `python -m <package_name> --help`.
 
+```{note}
+If you run the packaged project in the interactive environment like IPython or Databricks you can also consume the output of the `main()`
+which returns the `session.run()` output.
+```
+
+```python
+from spaceflights.__main__ import main
+
+def run_kedro_pipeline():
+   result = main(pipeline_name=<pipeline>)
+   do_something_with(<result>)
+```
+
 ### Docker, Airflow and other deployment targets
 
 There are various methods to deploy packaged pipelines via Kedro plugins:

diff --git a/kedro/framework/cli/project.py b/kedro/framework/cli/project.py
@@ -215,7 +215,7 @@ def run(  # noqa: PLR0913
     conf_source: str,
     params: dict[str, Any],
     namespace: str,
-) -> None:
+) -> dict[str, Any]:
     """Run the pipeline."""
 
     runner_obj = load_obj(runner or "SequentialRunner", "kedro.runner")
@@ -225,7 +225,7 @@ def run(  # noqa: PLR0913
     with KedroSession.create(
         env=env, conf_source=conf_source, extra_params=params
     ) as session:
-        session.run(
+        return session.run(
             tags=tuple_tags,
             runner=runner_obj(is_async=is_async),
             node_names=tuple_node_names,

diff --git a/...es/project/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/__main__.py b/...es/project/{{ cookiecutter.repo_name }}/src/{{ cookiecutter.python_package }}/__main__.py
@@ -3,20 +3,21 @@
 """
 import sys
 from pathlib import Path
+from typing import Any
 
 from kedro.framework.cli.utils import find_run_command
 from kedro.framework.project import configure_project
 
 
-def main(*args, **kwargs):
+def main(*args, **kwargs) -> Any:
     package_name = Path(__file__).parent.name
     configure_project(package_name)
 
     interactive = hasattr(sys, 'ps1')
     kwargs["standalone_mode"] = not interactive
 
     run = find_run_command(package_name)
-    run(*args, **kwargs)
+    return run(*args, **kwargs)
 
 
 if __name__ == "__main__":