diff --git a/docs/conf.py b/docs/conf.py index aac3d576..55379aab 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -56,6 +56,7 @@ autoapi_member_order = "bysource" html_theme = "sphinx_book_theme" - html_static_path = ["_static"] html_css_files = ["custom.css"] + +nbsphinx_allow_errors = True diff --git a/docs/tutorials/working_with_large_catalogs.ipynb b/docs/tutorials/working_with_large_catalogs.ipynb index 2b120186..01b92fe7 100644 --- a/docs/tutorials/working_with_large_catalogs.ipynb +++ b/docs/tutorials/working_with_large_catalogs.ipynb @@ -2,22 +2,420 @@ "cells": [ { "cell_type": "markdown", + "id": "cb1cb6a4-a0bc-44f2-ae48-d4de23e14c56", + "metadata": { + "tags": [] + }, + "source": [ + "# Working with large catalogs\n", + "\n", + "Large astronomical surveys contain a massive volume of data. Billion object, multi-terabyte sized catalogs are challenging to store and manipulate because they demand state-of-the-art hardware. Processing them is expensive, both in terms of runtime and memory consumption, and performing it in a single machine has become impractical. LSDB is a solution that enables scalable algorithm execution. It handles loading, querying, filtering and crossmatching astronomical data (of HiPSCat format) in a distributed environment. \n", + "\n", + "In this tutorial, we will demonstrate how to:\n", + "\n", + "1. Setup a Dask client for distributed processing\n", + "2. Load an object catalog with a set of desired columns\n", + "3. Select data from regions of the sky\n", + "4. Preview catalog data" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4bb2d314-32c8-4705-b980-f424ede22af1", + "metadata": {}, + "outputs": [], + "source": [ + "import lsdb" + ] + }, + { + "cell_type": "markdown", + "id": "20aa14ea", + "metadata": {}, + "source": [ + "## Installing dependencies\n", + "\n", + "To load catalogs from the web using the HTTP protocol we'll need to install __aiohttp__." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "d4b7cef3", + "metadata": {}, + "outputs": [], + "source": [ + "!pip install aiohttp --quiet" + ] + }, + { + "cell_type": "markdown", + "id": "c2606cc9-648a-49ac-bbe6-17dfe1f9309f", + "metadata": { + "tags": [] + }, + "source": [ + "## Creating a Dask client\n", + "\n", + "Dask is a framework that allows us to take advantage of distributed computing capabilities. \n", + "\n", + "With Dask, the operations defined in a workflow (e.g. this notebook) are added to a task graph which optimizes their order of execution. The operations are not immediately computed, that's for us to decide. As soon as we trigger the computations, Dask distributes the workload across its multiple workers and tasks are run efficiently in parallel. Usually the later we kick off the computations the better.\n", + "\n", + "Dask creates a client by default, if we do not instantiate one. If we do, we may, among others:\n", + "\n", + "- Specify the number of workers and the memory limit for each of them.\n", + "- Adjust the address for the dashboard to profile the operations while they run (by default it serves on port _8787_).\n", + "\n", + "For additional information please refer to https://distributed.dask.org/en/latest/client.html." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e16078b4-47b8-4939-83c4-1ad28bf1592e", + "metadata": {}, + "outputs": [], + "source": [ + "from dask.distributed import Client\n", + "\n", + "client = Client(n_workers=4, memory_limit=\"auto\")\n", + "client" + ] + }, + { + "cell_type": "markdown", + "id": "4051aff7-b09c-49a2-a8fd-fe32a1c2f34c", + "metadata": {}, + "source": [ + "## Loading a catalog\n", + "\n", + "We will load a small 5 degree radius cone from the `ZTF DR14` object catalog. \n", + "\n", + "Catalogs represent tabular data and are internally subdivided into partitions based on their positions in the sky. When processing a catalog, each worker is expected to be able to load a single partition at a time into memory for processing. Therefore, when loading a catalog, it's crucial to __specify the subset of columns__ we need for our science pipeline. Failure to specify these columns results in loading the entire partition table, which not only increases the usage of worker memory but also impacts runtime performance significantly." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ba169c5c", + "metadata": {}, + "outputs": [], + "source": [ + "surveys_path = \"https://epyc.astro.washington.edu/~lincc-frameworks/other_degree_surveys\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8520df60-167d-42f8-bc2c-e771d4ff75bd", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_path = f\"{surveys_path}/ztf/ztf_object\"\n", + "ztf_object = lsdb.read_hipscat(ztf_object_path, columns=[\"ps1_objid\", \"ra\", \"dec\"])\n", + "ztf_object" + ] + }, + { + "cell_type": "markdown", + "id": "9061e278-b55b-433a-af34-45b31d5295cc", + "metadata": {}, + "source": [ + "The catalog has been loaded lazily, we can see its metadata but no actual data is there yet. We will be defining more operations in this notebook. Only when we call `compute()` on the resulting catalog are operations executed, i.e. data is loaded from disk into memory for processing." + ] + }, + { + "cell_type": "markdown", + "id": "233abfb7-eb78-4d19-a133-1b17d22ff6c9", + "metadata": {}, + "source": [ + "## Selecting a region of the sky\n", + "\n", + "We may use 3 types of spatial filters (cone, polygon and box) to select a portion of the sky. \n", + "\n", + "Filtering consists of two main steps:\n", + "\n", + "- A __coarse__ stage, in which we find what pixels cover our desired region in the sky. These may overlap with the region and only be partially contained within the region boundaries. This means that some data points inside that pixel may fall outside of the region.\n", + "\n", + "- A __fine__ stage, where we filter the data points from each pixel to make sure they fall within the specified region.\n", + "\n", + "The `fine` parameter allows us to specify whether or not we desire to run the fine stage, for each search. It brings some overhead, so if your intention is to get a rough estimate of the data points for a region, you may disable it. It is always executed by default.\n", + "\n", + "```\n", + "catalog.box(..., fine=False)\n", + "catalog.cone_search(..., fine=False)\n", + "catalog.polygon_search(..., fine=False)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "bdd68cda", + "metadata": {}, + "source": [ + "Throughout this notebook we will use the Catalog's `plot_pixels` method to display the HEALPix of each resulting catalog as filters are applied." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "480b8473", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object.plot_pixels(\"ZTF_DR14 - pixel map\")" + ] + }, + { + "cell_type": "markdown", + "id": "83ad6987-8858-4f47-932d-a8dc240fd090", + "metadata": {}, + "source": [ + "### Cone search\n", + "\n", + "A cone search is defined by center `(ra, dec)`, in degrees, and radius `r`, in arcseconds." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c9460d44-fb4b-4768-8885-b57987e34dc3", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_cone = ztf_object.cone_search(ra=-60.3, dec=20.5, radius_arcsec=1 * 3600)\n", + "ztf_object_cone" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "4aa166cb-f5d6-4dc9-8d81-d6f6caceff40", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_cone.plot_pixels(\"ZTF_DR14 - cone pixel map\")" + ] + }, + { + "cell_type": "markdown", + "id": "9bb95bfe-2a37-4260-8d85-fc93cdcf03f2", + "metadata": {}, + "source": [ + "### Polygon search\n", + "\n", + "A polygon search is defined by convex polygon with vertices `[(ra1, dec1), (ra2, dec2)...]`, in degrees." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "25b7fea6-5af4-43e9-bd13-b83a4f0e095b", + "metadata": {}, + "outputs": [], + "source": [ + "vertices = [(-60.5, 15.1), (-62.5, 18.5), (-65.2, 15.3), (-64.2, 12.1)]\n", + "ztf_object_polygon = ztf_object.polygon_search(vertices)\n", + "ztf_object_polygon" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "51358523-b278-49c0-95c6-bb6574d5c2a7", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_polygon.plot_pixels(\"ZTF_DR14 - polygon pixel map\")" + ] + }, + { + "cell_type": "markdown", + "id": "2c7d0a1f-00df-4b79-82b7-fe2ba3d462b8", + "metadata": {}, + "source": [ + "### Box search\n", + "\n", + "A box search can be defined by:\n", + "\n", + "- Right ascension band `(ra1, ra2)`\n", + "- Declination band `(dec1, dec2)`\n", + "- Both right ascension and declination bands `[(ra1, ra2), (dec1, dec2)]`" + ] + }, + { + "cell_type": "markdown", + "id": "ca8c3815-0165-4123-8173-ccb6dfaa7eb5", + "metadata": {}, + "source": [ + "#### Right ascension band" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5e6ce24-34a1-42f5-99d6-be8ca0772113", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_box_ra = ztf_object.box(ra=[-65, -60])\n", + "ztf_object_box_ra" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "12edf29c-da39-47e5-9507-ebe4258ba789", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_box_ra.plot_pixels(\"ZTF_DR14 - RA band pixel map\")" + ] + }, + { + "cell_type": "markdown", + "id": "83c465ba-4548-4108-b9ec-5396ac760f64", + "metadata": {}, + "source": [ + "#### Declination band" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "03966efa-4e65-4aa8-ad1f-3e54bbf6a281", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_box_dec = ztf_object.box(dec=[12, 15])\n", + "ztf_object_box_dec" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e5de5b4a-78f6-4295-8c38-fc80a9a6f2a9", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_box_dec.plot_pixels(\"ZTF_DR14 - DEC band pixel map\")" + ] + }, + { + "cell_type": "markdown", + "id": "87952874-26fb-433f-adb7-28ac13c7cee3", + "metadata": { + "tags": [] + }, + "source": [ + "#### Right ascension and declination bands" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e58b4687-7ab6-4531-849a-c2f9ce35317b", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_box = ztf_object.box(ra=[-65, -60], dec=[12, 15])\n", + "ztf_object_box" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2392879a-31b8-45a3-a66a-56d02b2a47e7", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_box.plot_pixels(\"ZTF_DR14 - box pixel map\")" + ] + }, + { + "cell_type": "markdown", + "id": "9a887b31", + "metadata": {}, + "source": [ + "We can stack a several number of filters, which are applied in sequence. For example, `catalog.box().polygon_search()` should result in a perfectly valid HiPSCat catalog containing the objects that match both filters." + ] + }, + { + "cell_type": "markdown", + "id": "1594babc-7ce7-4b9c-ae63-424f3e863059", "metadata": {}, "source": [ - "# Working with large catalogs" + "## Previewing part of the data\n", + "\n", + "Computing an entire catalog requires loading all of its resulting data into memory, which is expensive and may lead to out-of-memory issues. \n", + "\n", + "Often our goal is to have a peek at a slice of data to make sure the workflow output is reasonable (e.g. to assess if some new created columns are present and their values have been properly processed). `head()` is a pandas-like method which allows us to preview part of the data for this purpose. It iterates over the existing catalog partitions, in sequence, and finds up to `n` number of rows.\n", + "\n", + "Notice that this method implicitly calls `compute()`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2d5fc9b4-169a-4478-846b-31dd50fba816", + "metadata": {}, + "outputs": [], + "source": [ + "ztf_object_cone.head()" ] }, { "cell_type": "markdown", + "id": "2233de79-1dc4-4737-a5a9-794e8f1c3d9b", + "metadata": {}, + "source": [ + "By default the first 5 rows of data will be shown but we can specify a higher number if we need." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "51bbd640-a1c9-4da2-a672-b18080e3dc27", "metadata": {}, - "source": [] + "outputs": [], + "source": [ + "ztf_object_cone.head(n=10)" + ] + }, + { + "cell_type": "markdown", + "id": "81bb9203-005e-4b68-b3f7-76225cbcbbcb", + "metadata": {}, + "source": [ + "## Closing the Dask client" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "32b90716-d4b3-4a51-8838-44af2ea89703", + "metadata": {}, + "outputs": [], + "source": [ + "client.close()" + ] } ], "metadata": { "language_info": { - "name": "python" + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.13" } }, "nbformat": 4, - "nbformat_minor": 2 + "nbformat_minor": 5 }