Skip to content

Commit

Permalink
Merge pull request #186 from lincc-frameworks/expand_quickstart
Browse files Browse the repository at this point in the history 
Expand quickstart.ipynb
  • Loading branch information
@jeremykubica
jeremykubica authored Jan 8, 2025
2 parents b60f6eb + d76bebc commit effbad0
Show file tree
Hide file tree
Showing 2 changed files with 304 additions and 34 deletions.
172 changes: 153 additions & 19 deletions docs/gettingstarted/quickstart.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,17 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"# Quickstart"
"# Quickstart\n",
"\n",
"This notebook provides a brief introduction to nested-pandas, including the motivation and basics for working with the data structure. For more in-depth descriptions, see the other tutorial notebooks."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Installation\n",
"\n",
"With a valid Python environment, nested-pandas and it's dependencies are easy to install using the `pip` package manager. The following command can be used to install it:"
]
},
Expand All @@ -27,7 +31,47 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Nested-Pandas is tailored towards efficient analysis of nested datasets. Let's load a toy dataset to show how it works."
"## Overview\n",
"\n",
"Nested-Pandas is tailored towards efficient analysis of nested data sets. This includes data that would normally be represented in a Pandas DataFrames with multiple rows needed to represent a single \"thing\" and therefor columns whose values will be identical for that item.\n",
"\n",
"As a concrete example, consider an astronomical data set storing information about observations of physical objects, such as stars and galaxies. One way to represent this in Pandas is to create one row per observation with an ID column indicating to which physical object the observation corresponds. However this approach ends up repeating a lot of data over each observation of the same object such as its location on the sky (RA, dec), its classification, etc. Further, any operations processing the data as time series requires the user to first perform a (potentially expensive) group-by operation to aggregate all of the data for each object.\n",
"\n",
"Let's create a flat pandas dataframe with three objects: object 0 has three observations, object 1 has three observations, and object 2 has 4 observations."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"# Represent nested time series information as a classic pandas dataframe.\n",
"my_data_frame = pd.DataFrame(\n",
" {\n",
" \"id\": [0, 0, 0, 1, 1, 1, 2, 2, 2, 2],\n",
" \"ra\": [10.0, 10.0, 10.0, 15.0, 15.0, 15.0, 12.1, 12.1, 12.1, 12.1],\n",
" \"dec\": [0.0, 0.0, 0.0, -1.0, -1.0, -1.0, 0.5, 0.5, 0.5, 0.5],\n",
" \"time\": [60676.0, 60677.0, 60678.0, 60675.0, 60676.5, 60677.0, 60676.6, 60676.7, 60676.8, 60676.9],\n",
" \"brightness\": [100.0, 101.0, 99.8, 5.0, 5.01, 4.98, 20.1, 20.5, 20.3, 20.2],\n",
" }\n",
")\n",
"my_data_frame"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that we cannot cleanly compress this by adding more columns (such as such as t0, t1, and so forth), because the number of observations is not bounded and may vary from object to object.\n",
"\n",
"Beyond astronomical data we might be interested in tracking patients blood pressure over time, the measure of intensities of emitted light at different wavelengths, or storing a list of the type of rock found at different depths of core samples. In each case it is possible to represent this data with multiple rows (such as one row for each patient + measurement pair) and associate them together by ids.\n",
"\n",
"Nested-pandas is designed for exactly this type of data by allowing columns to contain nested data. We can have regular columns with the (single) value for the objects’ unvarying characteristics (location on the sky, patentient birth date, location of the core sample) and nested columns for the values of each observation.\n",
"\n",
"Let's see an example:"
]
},
{
Expand All @@ -36,18 +80,50 @@
"metadata": {},
"outputs": [],
"source": [
"from nested_pandas.datasets import generate_data\n",
"from nested_pandas.nestedframe import NestedFrame\n",
"\n",
"# generate_data creates some toy data\n",
"nf = generate_data(10, 100) # 10 rows, 100 nested rows per row\n",
"# Create a nested data set\n",
"nf = NestedFrame.from_flat(\n",
" my_data_frame,\n",
" base_columns=[\"ra\", \"dec\"], # the columns not to nest\n",
" nested_columns=[\"time\", \"brightness\"], # the columns to nest\n",
" on=\"id\", # column used to associate rows\n",
" name=\"lightcurve\", # name of the nested column\n",
")\n",
"nf"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The above dataframe is a `NestedFrame`, which extends the capabilities of the Pandas `DataFrame` to support columns with nested information. In this example, we have the top level dataframe with 10 rows and 2 typical columns, \"a\" and \"b\". The \"nested\" column contains a dataframe in each row. We can inspect the contents of the \"nested\" column using pandas API tooling like `loc`."
"The above dataframe is a `NestedFrame`, which extends the capabilities of the Pandas `DataFrame` to support columns with nested information. \n",
"\n",
"We now have the top level dataframe with 3 rows, each of which corresponds to a single object. The table has three columns beyond \"id\". Two columns, \"ra\" and \"dec\", have a single value for the object (in this case the position on the sky). The last column \"lightcurve\" contains a nested table with a series of observation times and observation brightnesses for the object. As we will see below, this nested table allows the user to easily access to the all of the observations for a given object.\n",
"\n",
"## Accessing Nested Data\n",
"\n",
"We can inspect the contents of the \"lightcurve\" column using pandas API tooling like `loc`."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nf.loc[0][\"lightcurve\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we see that within the \"lightcurve\" column there are tables with their own data. In this case we have 2 columns (\"time\" and \"brightness\") that represent a time series of observations. \n",
"\n",
"Note that `loc` itself accesses the row, so the combination of `nf.loc[0][\"lightcurve\"]` means we are looking at value in the \"lightcurve\" column for a single row (row 0). If we just use `nf.loc[0]` we would retrieve the entire row, including the nested \"lightcurve\" column and all other columns. Similarly if we use `nf[\"lightcurve]` we retrieve the nested column for all rows. What makes the nesting useful is that once we access the nested entry for a specific row, we can treat the value as a table in its own right.\n",
"\n",
"As in Pandas, we can still access individual entries from a column based on the row index. Thus we can access the values (in a table) in row 0 of the nested column as `nf[\"lightcurve\"][0]` as well."
]
},
{
Expand All @@ -56,14 +132,68 @@
"metadata": {},
"outputs": [],
"source": [
"nf.loc[0][\"nested\"]"
"nf[\"lightcurve\"][0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here we see that within the \"nested\" column there are `NestedFrame` objects with their own data. In this case we have 3 columns (\"t\", \"flux\", and \"band\"). Alternatively, we could inspect the available columns using some custom properties of the `NestedFrame`."
"We can also use dot notation to access all the values in a nested sub column:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nf[\"lightcurve.time\"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that \"lightcurve.time\" contains the time values for all rows, but also preserves the nesting information. The id column of the returned data maps the top-level row (in `nf`) to where this value resides.\n",
"\n",
"Similarly, we can access the values for a given top-level row by index. To get all the `time` values for row 0 we could specify:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nf[\"lightcurve.time\"][0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here the `[0]` is telling our nested frame to access the values of the series `nf[\"lightcurve.time\"]` where the id = 0. If we try `nf[\"lightcurve.time\"][0][0]` we again match id = 0 and return the same frame. \n",
"\n",
"To access a single element within the series, we need to use its location:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"nf[\"lightcurve.time\"][0].iloc[0]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Inspecting Nested Frames\n",
"\n",
"We can inspect the available columns using some custom properties of the `NestedFrame`."
]
},
{
Expand All @@ -90,7 +220,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"nested-pandas extends the Pandas API, meaning any operation you could do in Pandas is available within nested-pandas. However, nested-pandas has additional functionality and tooling to better support working with nested datasets. For example, let's look at `query`:"
"## Pandas Operations\n",
"\n",
"Nested-pandas extends the Pandas API, meaning any operation you could do in Pandas is available within nested-pandas. However, nested-pandas has additional functionality and tooling to better support working with nested datasets. For example, let's look at `query`:"
]
},
{
Expand All @@ -100,7 +232,7 @@
"outputs": [],
"source": [
"# Normal queries work as expected, rejecting rows from the dataframe that don't meet the criteria\n",
"nf.query(\"a > 0.2\")"
"nf.query(\"ra > 11.2\")"
]
},
{
Expand All @@ -116,8 +248,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Applies the query to \"nested\", filtering based on \"t >17\"\n",
"nf_g = nf.query(\"nested.t > 17.0\")\n",
"# Applies the query to \"nested\", filtering based on \"time > 60676.0\"\n",
"nf_g = nf.query(\"lightcurve.time > 60676.0\")\n",
"nf_g"
]
},
Expand All @@ -134,8 +266,8 @@
"metadata": {},
"outputs": [],
"source": [
"# All t <= 17.0 have been removed\n",
"nf_g.loc[0][\"nested\"]"
"# All t <= 60676.0 have been removed\n",
"nf_g.loc[0][\"lightcurve\"]"
]
},
{
Expand All @@ -149,6 +281,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Reduce Function\n",
"\n",
"Finally, we'll end with the flexible `reduce` function. `reduce` functions similarly to Pandas' `apply` but flattens (reduces) the inputs from nested layers into array inputs to the given apply function. For example, let's find the mean flux for each dataframe in \"nested\":"
]
},
Expand All @@ -162,7 +296,7 @@
"\n",
"# use hierarchical column names to access the flux column\n",
"# passed as an array to np.mean\n",
"nf.reduce(np.mean, \"nested.flux\")"
"nf.reduce(np.mean, \"lightcurve.brightness\")"
]
},
{
Expand All @@ -186,7 +320,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Applying some inputs via reduce, we see how it sends inputs to a given function."
"Applying some inputs via reduce, we see how it sends inputs to a given function. The output frame `nf_inputs` consists of two columns containing the output of the “ra” column and the “lightcurve.time” column."
]
},
{
Expand All @@ -195,7 +329,7 @@
"metadata": {},
"outputs": [],
"source": [
"nf_inputs = nf.reduce(show_inputs, \"a\", \"nested.band\")\n",
"nf_inputs = nf.reduce(show_inputs, \"ra\", \"lightcurve.time\")\n",
"nf_inputs"
]
},
Expand All @@ -211,7 +345,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "nested",
"language": "python",
"name": "python3"
},
Expand All @@ -225,7 +359,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.11"
"version": "3.10.4"
}
},
"nbformat": 4,
Expand Down
Loading

0 comments on commit effbad0

Please sign in to comment.