Skip to content

Commit

Permalink
[doc_push][main] built from caba46a (main). Redirects: main -> 0.7.0d…
Browse files Browse the repository at this point in the history 
…ev0.
  • Loading branch information
TorchX CI Runner committed Nov 20, 2023
1 parent 23eeae0 commit 2ab0548
Show file tree
Hide file tree
Showing 12 changed files with 576 additions and 528 deletions.
Binary file modified 0.7.0dev0/_downloads/00d67ca99480eb8a810baf2dcc8feeff/examples_apps_python.zip
View file
Binary file not shown.
Binary file modified 0.7.0dev0/_downloads/0565b2ee0f8662d1acf800c01d94e038/examples_apps_jupyter.zip
View file
Binary file not shown.
Binary file modified 0.7.0dev0/_downloads/3d78ac59c117d4c58eca135c8ee3f34a/examples_pipelines_python.zip
View file
Binary file not shown.
Binary file modified 0.7.0dev0/_downloads/7e7e4caf540a83f1776a6e62b5c440ff/examples_pipelines_jupyter.zip
View file
Binary file not shown.
68 changes: 54 additions & 14 deletions 0.7.0dev0/_modules/torchx/specs/named_resources_aws.html
View file

Large diffs are not rendered by default.

243 changes: 122 additions & 121 deletions 0.7.0dev0/custom_components.html
View file

Large diffs are not rendered by default.

375 changes: 191 additions & 184 deletions 0.7.0dev0/custom_components.ipynb
View file

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions 0.7.0dev0/pipelines/airflow.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -490,26 +490,26 @@ <h1>Airflow<a class="headerlink" href="#Airflow" title="Permalink to this headin
<div class="prompt empty docutils container">
</div>
<div class="output_area rendered_html docutils container">
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #808000; text-decoration-color: #808000; font-weight: bold">/tmp/ipykernel_4112/454499020.py:</span><span style="color: #808000; text-decoration-color: #808000; font-weight: bold">3</span><span style="color: #808000; text-decoration-color: #808000; font-weight: bold"> RemovedInAirflow3Warning</span><span style="color: #808000; text-decoration-color: #808000">: Param `schedule_interval` is deprecated and will be removed in a future release. Please use `schedule` instead.</span>
<pre style="white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace"><span style="color: #808000; text-decoration-color: #808000; font-weight: bold">/tmp/ipykernel_4138/454499020.py:</span><span style="color: #808000; text-decoration-color: #808000; font-weight: bold">3</span><span style="color: #808000; text-decoration-color: #808000; font-weight: bold"> RemovedInAirflow3Warning</span><span style="color: #808000; text-decoration-color: #808000">: Param `schedule_interval` is deprecated and will be removed in a future release. Please use `schedule` instead.</span>
</pre></div>
</div>
<div class="nboutput nblast docutils container">
<div class="prompt empty docutils container">
</div>
<div class="output_area docutils container">
<div class="highlight"><pre>
[<span class="ansi-blue-fg">2023-11-15T20:30:01.449+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1159} INFO - Dependencies all met for dep_context=non-requeueable deps ti=&lt;TaskInstance: example_python_operator-p5x25340v0flr.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]&gt;
[<span class="ansi-blue-fg">2023-11-15T20:30:01.455+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1159} INFO - Dependencies all met for dep_context=requeueable deps ti=&lt;TaskInstance: example_python_operator-p5x25340v0flr.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]&gt;
[<span class="ansi-blue-fg">2023-11-15T20:30:01.456+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1361} INFO - Starting attempt 1 of 1
[<span class="ansi-blue-fg">2023-11-15T20:30:01.457+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1430} WARNING - cannot record queued_duration for task hello_torchx because previous state change time has not been saved
[<span class="ansi-blue-fg">2023-11-15T20:30:01.466+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1382} INFO - Executing &lt;Task(_PythonDecoratedOperator): hello_torchx&gt; on 2021-09-13 00:00:00+00:00
[<span class="ansi-blue-fg">2023-11-15T20:30:01.706+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1662} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER=&#39;airflow&#39; AIRFLOW_CTX_DAG_ID=&#39;example_python_operator-p5x25340v0flr&#39; AIRFLOW_CTX_TASK_ID=&#39;hello_torchx&#39; AIRFLOW_CTX_EXECUTION_DATE=&#39;2021-09-13T00:00:00+00:00&#39; AIRFLOW_CTX_TRY_NUMBER=&#39;1&#39; AIRFLOW_CTX_DAG_RUN_ID=&#39;manual__2021-09-13T00:00:00+00:00&#39;
[<span class="ansi-blue-fg">2023-11-15T20:30:02.346+0000</span>] {<span class="ansi-blue-fg">api.py:</span>70} INFO - Tracker configurations: {}
[<span class="ansi-blue-fg">2023-11-15T20:30:02.350+0000</span>] {<span class="ansi-blue-fg">local_scheduler.py:</span>716} INFO - Log directory not set in scheduler cfg. Creating a temporary log dir that will be deleted on exit. To preserve log directory set the `log_dir` cfg option
[<span class="ansi-blue-fg">2023-11-15T20:30:02.351+0000</span>] {<span class="ansi-blue-fg">local_scheduler.py:</span>722} INFO - Log directory is: /tmp/torchx_3wr3uuvk
[<span class="ansi-blue-fg">2023-11-20T18:06:46.244+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1159} INFO - Dependencies all met for dep_context=non-requeueable deps ti=&lt;TaskInstance: example_python_operator-h6k13r7rqcpb0c.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]&gt;
[<span class="ansi-blue-fg">2023-11-20T18:06:46.250+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1159} INFO - Dependencies all met for dep_context=requeueable deps ti=&lt;TaskInstance: example_python_operator-h6k13r7rqcpb0c.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]&gt;
[<span class="ansi-blue-fg">2023-11-20T18:06:46.250+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1361} INFO - Starting attempt 1 of 1
[<span class="ansi-blue-fg">2023-11-20T18:06:46.251+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1430} WARNING - cannot record queued_duration for task hello_torchx because previous state change time has not been saved
[<span class="ansi-blue-fg">2023-11-20T18:06:46.261+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1382} INFO - Executing &lt;Task(_PythonDecoratedOperator): hello_torchx&gt; on 2021-09-13 00:00:00+00:00
[<span class="ansi-blue-fg">2023-11-20T18:06:46.522+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1662} INFO - Exporting env vars: AIRFLOW_CTX_DAG_OWNER=&#39;airflow&#39; AIRFLOW_CTX_DAG_ID=&#39;example_python_operator-h6k13r7rqcpb0c&#39; AIRFLOW_CTX_TASK_ID=&#39;hello_torchx&#39; AIRFLOW_CTX_EXECUTION_DATE=&#39;2021-09-13T00:00:00+00:00&#39; AIRFLOW_CTX_TRY_NUMBER=&#39;1&#39; AIRFLOW_CTX_DAG_RUN_ID=&#39;manual__2021-09-13T00:00:00+00:00&#39;
[<span class="ansi-blue-fg">2023-11-20T18:06:47.151+0000</span>] {<span class="ansi-blue-fg">api.py:</span>70} INFO - Tracker configurations: {}
[<span class="ansi-blue-fg">2023-11-20T18:06:47.155+0000</span>] {<span class="ansi-blue-fg">local_scheduler.py:</span>716} INFO - Log directory not set in scheduler cfg. Creating a temporary log dir that will be deleted on exit. To preserve log directory set the `log_dir` cfg option
[<span class="ansi-blue-fg">2023-11-20T18:06:47.156+0000</span>] {<span class="ansi-blue-fg">local_scheduler.py:</span>722} INFO - Log directory is: /tmp/torchx_t4_ggmjc
Hello, TorchX!
[<span class="ansi-blue-fg">2023-11-15T20:30:02.458+0000</span>] {<span class="ansi-blue-fg">python.py:</span>194} INFO - Done. Returned value was: None
[<span class="ansi-blue-fg">2023-11-15T20:30:02.463+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1400} INFO - Marking task as SUCCESS. dag_id=example_python_operator-p5x25340v0flr, task_id=hello_torchx, execution_date=20210913T000000, start_date=20231115T203001, end_date=20231115T203002
[<span class="ansi-blue-fg">2023-11-20T18:06:47.261+0000</span>] {<span class="ansi-blue-fg">python.py:</span>194} INFO - Done. Returned value was: None
[<span class="ansi-blue-fg">2023-11-20T18:06:47.266+0000</span>] {<span class="ansi-blue-fg">taskinstance.py:</span>1400} INFO - Marking task as SUCCESS. dag_id=example_python_operator-h6k13r7rqcpb0c, task_id=hello_torchx, execution_date=20210913T000000, start_date=20231120T180646, end_date=20231120T180647
</pre></div></div>
</div>
<p>If all goes well you should see <code class="docutils literal notranslate"><span class="pre">Hello,</span> <span class="pre">TorchX!</span></code> printed above.</p>
Expand Down
64 changes: 32 additions & 32 deletions 0.7.0dev0/pipelines/airflow.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"cells": [
{
"cell_type": "markdown",
"id": "93a7861f",
"id": "1f7905f1",
"metadata": {},
"source": [
"# Airflow\n",
Expand All @@ -18,13 +18,13 @@
{
"cell_type": "code",
"execution_count": 1,
"id": "2fc8c265",
"id": "a8d5f004",
"metadata": {
"execution": {
"iopub.execute_input": "2023-11-15T20:30:00.689299Z",
"iopub.status.busy": "2023-11-15T20:30:00.689099Z",
"iopub.status.idle": "2023-11-15T20:30:01.227261Z",
"shell.execute_reply": "2023-11-15T20:30:01.226667Z"
"iopub.execute_input": "2023-11-20T18:06:45.478072Z",
"iopub.status.busy": "2023-11-20T18:06:45.477880Z",
"iopub.status.idle": "2023-11-20T18:06:46.025042Z",
"shell.execute_reply": "2023-11-20T18:06:46.024394Z"
}
},
"outputs": [],
Expand All @@ -44,7 +44,7 @@
},
{
"cell_type": "markdown",
"id": "2bbf48e7",
"id": "b09eb509",
"metadata": {},
"source": [
"To launch a TorchX job from Airflow you can create a Airflow Python task to\n",
Expand All @@ -56,13 +56,13 @@
{
"cell_type": "code",
"execution_count": 2,
"id": "45a7ff2e",
"id": "9a5e91b5",
"metadata": {
"execution": {
"iopub.execute_input": "2023-11-15T20:30:01.230419Z",
"iopub.status.busy": "2023-11-15T20:30:01.230080Z",
"iopub.status.idle": "2023-11-15T20:30:01.235036Z",
"shell.execute_reply": "2023-11-15T20:30:01.234492Z"
"iopub.execute_input": "2023-11-20T18:06:46.028464Z",
"iopub.status.busy": "2023-11-20T18:06:46.027823Z",
"iopub.status.idle": "2023-11-20T18:06:46.032744Z",
"shell.execute_reply": "2023-11-20T18:06:46.032089Z"
}
},
"outputs": [],
Expand Down Expand Up @@ -93,7 +93,7 @@
},
{
"cell_type": "markdown",
"id": "56fbe9e6",
"id": "7ff39e97",
"metadata": {},
"source": [
"Once we have the task defined we can put it into a Airflow DAG and run it like\n",
Expand All @@ -103,24 +103,24 @@
{
"cell_type": "code",
"execution_count": 3,
"id": "fbcb395b",
"id": "13062fd5",
"metadata": {
"execution": {
"iopub.execute_input": "2023-11-15T20:30:01.237378Z",
"iopub.status.busy": "2023-11-15T20:30:01.237182Z",
"iopub.status.idle": "2023-11-15T20:30:02.471498Z",
"shell.execute_reply": "2023-11-15T20:30:02.470791Z"
"iopub.execute_input": "2023-11-20T18:06:46.035373Z",
"iopub.status.busy": "2023-11-20T18:06:46.034991Z",
"iopub.status.idle": "2023-11-20T18:06:47.274131Z",
"shell.execute_reply": "2023-11-20T18:06:47.273564Z"
}
},
"outputs": [
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">/tmp/ipykernel_4112/454499020.py:</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">3</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> RemovedInAirflow3Warning</span><span style=\"color: #808000; text-decoration-color: #808000\">: Param `schedule_interval` is deprecated and will be removed in a future release. Please use `schedule` instead.</span>\n",
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">/tmp/ipykernel_4138/454499020.py:</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">3</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\"> RemovedInAirflow3Warning</span><span style=\"color: #808000; text-decoration-color: #808000\">: Param `schedule_interval` is deprecated and will be removed in a future release. Please use `schedule` instead.</span>\n",
"</pre>\n"
],
"text/plain": [
"\u001b[1;33m/tmp/ipykernel_4112/\u001b[0m\u001b[1;33m454499020.py\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m3\u001b[0m\u001b[1;33m RemovedInAirflow3Warning\u001b[0m\u001b[33m: Param `schedule_interval` is deprecated and will be removed in a future release. Please use `schedule` instead.\u001b[0m\n"
"\u001b[1;33m/tmp/ipykernel_4138/\u001b[0m\u001b[1;33m454499020.py\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m3\u001b[0m\u001b[1;33m RemovedInAirflow3Warning\u001b[0m\u001b[33m: Param `schedule_interval` is deprecated and will be removed in a future release. Please use `schedule` instead.\u001b[0m\n"
]
},
"metadata": {},
Expand All @@ -130,78 +130,78 @@
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:01.449+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1159} INFO\u001b[0m - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: example_python_operator-p5x25340v0flr.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]>\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:46.244+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1159} INFO\u001b[0m - Dependencies all met for dep_context=non-requeueable deps ti=<TaskInstance: example_python_operator-h6k13r7rqcpb0c.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]>\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:01.455+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1159} INFO\u001b[0m - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: example_python_operator-p5x25340v0flr.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]>\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:46.250+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1159} INFO\u001b[0m - Dependencies all met for dep_context=requeueable deps ti=<TaskInstance: example_python_operator-h6k13r7rqcpb0c.hello_torchx manual__2021-09-13T00:00:00+00:00 [None]>\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:01.456+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1361} INFO\u001b[0m - Starting attempt 1 of 1\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:46.250+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1361} INFO\u001b[0m - Starting attempt 1 of 1\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:01.457+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1430} WARNING\u001b[0m - cannot record queued_duration for task hello_torchx because previous state change time has not been saved\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:46.251+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1430} WARNING\u001b[0m - cannot record queued_duration for task hello_torchx because previous state change time has not been saved\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:01.466+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1382} INFO\u001b[0m - Executing <Task(_PythonDecoratedOperator): hello_torchx> on 2021-09-13 00:00:00+00:00\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:46.261+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1382} INFO\u001b[0m - Executing <Task(_PythonDecoratedOperator): hello_torchx> on 2021-09-13 00:00:00+00:00\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:01.706+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1662} INFO\u001b[0m - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='example_python_operator-p5x25340v0flr' AIRFLOW_CTX_TASK_ID='hello_torchx' AIRFLOW_CTX_EXECUTION_DATE='2021-09-13T00:00:00+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2021-09-13T00:00:00+00:00'\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:46.522+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1662} INFO\u001b[0m - Exporting env vars: AIRFLOW_CTX_DAG_OWNER='airflow' AIRFLOW_CTX_DAG_ID='example_python_operator-h6k13r7rqcpb0c' AIRFLOW_CTX_TASK_ID='hello_torchx' AIRFLOW_CTX_EXECUTION_DATE='2021-09-13T00:00:00+00:00' AIRFLOW_CTX_TRY_NUMBER='1' AIRFLOW_CTX_DAG_RUN_ID='manual__2021-09-13T00:00:00+00:00'\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:02.346+0000\u001b[0m] {\u001b[34mapi.py:\u001b[0m70} INFO\u001b[0m - Tracker configurations: {}\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:47.151+0000\u001b[0m] {\u001b[34mapi.py:\u001b[0m70} INFO\u001b[0m - Tracker configurations: {}\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:02.350+0000\u001b[0m] {\u001b[34mlocal_scheduler.py:\u001b[0m716} INFO\u001b[0m - Log directory not set in scheduler cfg. Creating a temporary log dir that will be deleted on exit. To preserve log directory set the `log_dir` cfg option\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:47.155+0000\u001b[0m] {\u001b[34mlocal_scheduler.py:\u001b[0m716} INFO\u001b[0m - Log directory not set in scheduler cfg. Creating a temporary log dir that will be deleted on exit. To preserve log directory set the `log_dir` cfg option\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:02.351+0000\u001b[0m] {\u001b[34mlocal_scheduler.py:\u001b[0m722} INFO\u001b[0m - Log directory is: /tmp/torchx_3wr3uuvk\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:47.156+0000\u001b[0m] {\u001b[34mlocal_scheduler.py:\u001b[0m722} INFO\u001b[0m - Log directory is: /tmp/torchx_t4_ggmjc\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Hello, TorchX!\n",
"[\u001b[34m2023-11-15T20:30:02.458+0000\u001b[0m] {\u001b[34mpython.py:\u001b[0m194} INFO\u001b[0m - Done. Returned value was: None\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:47.261+0000\u001b[0m] {\u001b[34mpython.py:\u001b[0m194} INFO\u001b[0m - Done. Returned value was: None\u001b[0m\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\u001b[34m2023-11-15T20:30:02.463+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1400} INFO\u001b[0m - Marking task as SUCCESS. dag_id=example_python_operator-p5x25340v0flr, task_id=hello_torchx, execution_date=20210913T000000, start_date=20231115T203001, end_date=20231115T203002\u001b[0m\n"
"[\u001b[34m2023-11-20T18:06:47.266+0000\u001b[0m] {\u001b[34mtaskinstance.py:\u001b[0m1400} INFO\u001b[0m - Marking task as SUCCESS. dag_id=example_python_operator-h6k13r7rqcpb0c, task_id=hello_torchx, execution_date=20210913T000000, start_date=20231120T180646, end_date=20231120T180647\u001b[0m\n"
]
}
],
Expand Down Expand Up @@ -233,7 +233,7 @@
},
{
"cell_type": "markdown",
"id": "a684a859",
"id": "09b5ab65",
"metadata": {},
"source": [
"If all goes well you should see `Hello, TorchX!` printed above.\n",
Expand Down
Loading

0 comments on commit 2ab0548

Please sign in to comment.