3-task-graphs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "39588e45-c712-440c-8ee6-db9a04e0f4dc",
   "metadata": {},
   "source": [
    "<img src=\"images/dask_horizontal.svg\"\n",
    "     width=\"300px\"\n",
    "     alt=\"Dask logo\">\n",
    "     \n",
    "# Understanding the Task Graph\n",
    "\n",
    "Everything operation in Dask whether it's dataframe or array or delayed, creates some **tasks**. Each task knows its dependencies so Dask can create a graph of all the tasks and which depend on which. Put another way: a task graph is a **DAG** (Directed Acyclyic Graph). Read all about [task graphs](https://docs.dask.org/en/latest/spec.html) and [high level graphs](https://docs.dask.org/en/latest/high-level-graphs.html).\n",
    "\n",
    "<img src=\"images/dask-task-graphs.png\" width=\"100%\"> "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35a58eee-71c2-4ec8-af76-8514bc6062bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.array as da\n",
    "\n",
    "arr = da.random.random(size=(1_000, 1_000), chunks=(250, 250))\n",
    "arr.dask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bb50d0ec-630d-4b9e-8208-e998eeabdfc9",
   "metadata": {},
   "source": [
    "`arr` is itself a graph, and doing computations on `arr` produces another graph."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efe92e7f-32b0-4751-adcf-3b7d670f2794",
   "metadata": {},
   "outputs": [],
   "source": [
    "result = (arr + arr.T).sum(axis=0).mean()\n",
    "result.dask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "126da117-1689-4b73-a8fd-fe369fb8224f",
   "metadata": {},
   "source": [
    "For small enough graphs, we can get a visual representation of the graph. Each layer in the drawing of the task graph (below) corresponds to a layer in the high level graph (above)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6f567be-304a-406c-b26e-f08f09b1a701",
   "metadata": {},
   "outputs": [],
   "source": [
    "result.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b544cd38-66a7-487d-bcaf-798a0ee2ef05",
   "metadata": {},
   "source": [
    "Once the graph is constructed it goes through **optimization**, **scheduling**, and **execution**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9901a7f3-cc9e-4b06-8426-1def070a8dcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "result.visualize(optimize_graph=True, color=\"order\", node_attr={\"penwidth\": \"4\"})"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}