Make examples consistent

docs/gettingstarted/quickstart.ipynb

@@ -33,9 +33,11 @@
    "source": [
     "## Overview\n",
     "\n",
-    "Nested-Pandas is tailored towards efficient analysis of nested data sets. This includes data that would normally be represented in a Pandas DataFrames with multiple rows needed to represent a single \"thing\" and therefor multiple columns whose values will be identical for that item.\n",
+    "Nested-Pandas is tailored towards efficient analysis of nested data sets. This includes data that would normally be represented in a Pandas DataFrames with multiple rows needed to represent a single \"thing\" and therefor columns whose values will be identical for that item.\n",
     "\n",
-    "As a concrete example, consider an astronomical data set storing information about observations of physical objects, such as stars and galaxies. One way to represent this in Pandas is to create one row per observation with an ID column indicating to which physical object the observation corresponds. However this approach ends up repeating a lot of data over each observation of the same object such as its location on the sky (RA, dec), its classification, etc. Further any operations processing the data as time series requires the user to first perform a (potentially expensive) group-by operation."
+    "As a concrete example, consider an astronomical data set storing information about observations of physical objects, such as stars and galaxies. One way to represent this in Pandas is to create one row per observation with an ID column indicating to which physical object the observation corresponds. However this approach ends up repeating a lot of data over each observation of the same object such as its location on the sky (RA, dec), its classification, etc. Further, any operations processing the data as time series requires the user to first perform a (potentially expensive) group-by operation to aggregate all of the data for each object.\n",
+    "\n",
+    "Let's create a flat pandas dataframe with three objects: object 0 has three observations, object 1 has three observations, and object 2 has 4 observations."
    ]
   },
   {
@@ -49,11 +51,11 @@
     "# Represent nested time series information as a classic pandas dataframe.\n",
     "my_data_frame = pd.DataFrame(\n",
     "    {\n",
-    "        \"id\": [0, 0, 0, 1, 1],\n",
-    "        \"ra\": [10.0, 10.0, 10.0, 15.0, 15.0],\n",
-    "        \"dec\": [0.0, 0.0, 0.0, -1.0, -1.0],\n",
-    "        \"time\": [60676.0, 60677.0, 60678.0, 60675.0, 60676.5],\n",
-    "        \"brightness\": [100.0, 101.0, 99.8, 5.0, 5.01],\n",
+    "        \"id\": [0, 0, 0, 1, 1, 1, 2, 2, 2, 2],\n",
+    "        \"ra\": [10.0, 10.0, 10.0, 15.0, 15.0, 15.0, 12.1, 12.1, 12.1, 12.1],\n",
+    "        \"dec\": [0.0, 0.0, 0.0, -1.0, -1.0, -1.0, 0.5, 0.5, 0.5, 0.5],\n",
+    "        \"time\": [60676.0, 60677.0, 60678.0, 60675.0, 60676.5, 60677.0, 60676.6, 60676.7, 60676.8, 60676.9],\n",
+    "        \"brightness\": [100.0, 101.0, 99.8, 5.0, 5.01, 4.98, 20.1, 20.5, 20.3, 20.2],\n",
     "    }\n",
     ")\n",
     "my_data_frame"
@@ -63,9 +65,11 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Beyond astronomical data we might be interested in tracking a patients blood pressure over time, a measure of intensities at different wavelengths, or storing a list of the type of rock found at different depths of core samples. In each case it is possible to represent this data with multiple rows (one row for each patient + measurement pair) and associate them together by ids.\n",
+    "Note that we cannot cleanly compress this by adding more columns (such as such as t0, t1, and so forth), because the number of observations is not bounded and may vary from object to object.\n",
+    "\n",
+    "Beyond astronomical data we might be interested in tracking patients blood pressure over time, the measure of intensities of emitted light at different wavelengths, or storing a list of the type of rock found at different depths of core samples. In each case it is possible to represent this data with multiple rows (such as one row for each patient + measurement pair) and associate them together by ids.\n",
     "\n",
-    "In contrast, nested-pandas allows columns to represent nested data. We can have columns with the (single) value for the objects’ unvarying characteristics (location on the sky, patentient birth date, location of the core sample) and nested columns for the values of each observation.\n",
+    "Nested-pandas is designed for exactly this type of data by allowing columns to contain nested data. We can have regular columns with the (single) value for the objects’ unvarying characteristics (location on the sky, patentient birth date, location of the core sample) and nested columns for the values of each observation.\n",
     "\n",
     "Let's see an example:"
    ]
@@ -76,10 +80,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from nested_pandas.datasets import generate_data\n",
+    "from nested_pandas.nestedframe import NestedFrame\n",
     "\n",
-    "# generate_data creates some toy data\n",
-    "nf = generate_data(10, 100)  # 10 rows, 100 nested rows per row\n",
+    "# Create a nested data set\n",
+    "nf = NestedFrame.from_flat(\n",
+    "    my_data_frame,\n",
+    "    base_columns=[\"ra\", \"dec\"],  # the columns not to nest\n",
+    "    nested_columns=[\"time\", \"brightness\"],  # the columns to nest\n",
+    "    on=\"id\",  # column used to associate rows\n",
+    "    name=\"lightcurve\",  # name of the nested column\n",
+    ")\n",
     "nf"
    ]
   },
@@ -89,11 +99,11 @@
    "source": [
     "The above dataframe is a `NestedFrame`, which extends the capabilities of the Pandas `DataFrame` to support columns with nested information. \n",
     "\n",
-    "In this example, we have the top level dataframe with 10 rows and 2 typical columns, \"a\" and \"b\".  These represent object level attributes. The \"nested\" column contains a dataframe in each row that represents a time series (or observation level values).  As we will see below, this allows easy access to the all of the observations for a given object.\n",
+    "We now have the top level dataframe with 3 rows, each of which corresponds to a single object. The table has three columns beyond \"id\". Two columns, \"ra\" and \"dec\", have a single value for the object (in this case the position on the sky). The last column \"lightcurve\" contains a nested table with a series of observation times and observation brightnesses for the object. As we will see below, this nested table allows the user to easily access to the all of the observations for a given object.\n",
     "\n",
     "## Accessing Nested Data\n",
     "\n",
-    "We can inspect the contents of the \"nested\" column using pandas API tooling like `loc`."
+    "We can inspect the contents of the \"lightcurve\" column using pandas API tooling like `loc`."
    ]
   },
   {
@@ -102,18 +112,18 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "nf.loc[0][\"nested\"]"
+    "nf.loc[0][\"lightcurve\"]"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Here we see that within the \"nested\" column there are `NestedFrame` objects with their own data. In this case we have 3 columns (\"t\", \"flux\", and \"band\") that represent a time series of observations. \n",
+    "Here we see that within the \"lightcurve\" column there are tables with their own data. In this case we have 2 columns (\"time\" and \"brightness\") that represent a time series of observations. \n",
     "\n",
-    "Note that `loc` itself accesses the row, so the combination of `nf.loc[0][\"nested\"]` means we are looking at value in the \"nested\" column for a single row (row 0). If we just use `nf.loc[0]` we would retrieve the entire row, including the nested colum and all other columns. Similarly if we use `nf[\"nested”]` we retrieve the nested column for all rows. What makes the nesting useful is that once we access the nested entry for a specific row, we can treat the value as a table in its own right.\n",
+    "Note that `loc` itself accesses the row, so the combination of `nf.loc[0][\"lightcurve\"]` means we are looking at value in the \"lightcurve\" column for a single row (row 0). If we just use `nf.loc[0]` we would retrieve the entire row, including the nested \"lightcurve\" column and all other columns. Similarly if we use `nf[\"lightcurve]` we retrieve the nested column for all rows. What makes the nesting useful is that once we access the nested entry for a specific row, we can treat the value as a table in its own right.\n",
     "\n",
-    "As in Pandas, we can still access individual entries from a column ased on the row index. Thus we can access the entry (table) in row 0 of the nested column as `nf[\"nested\"][0]` as well."
+    "As in Pandas, we can still access individual entries from a column based on the row index. Thus we can access the values (in a table) in row 0 of the nested column as `nf[\"lightcurve\"][0]` as well."
    ]
   },
   {
@@ -122,7 +132,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "nf[\"nested\"][0]"
+    "nf[\"lightcurve\"][0]"
    ]
   },
   {
@@ -138,16 +148,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "nf[\"nested.t\"]"
+    "nf[\"lightcurve.time\"]"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Note that \"nested.t\" contains the t values for all rows, but preserves the nesting information. The id column of the returned data maps the top-level row (in `nf`) whether this value resides.\n",
+    "Note that \"lightcurve.time\" contains the time values for all rows, but also preserves the nesting information. The id column of the returned data maps the top-level row (in `nf`) to where this value resides.\n",
     "\n",
-    "Similarly we can access the values for a given top-level row by index. To get all the `t` values for row 0 we could specify:"
+    "Similarly, we can access the values for a given top-level row by index. To get all the `time` values for row 0 we could specify:"
    ]
   },
   {
@@ -156,14 +166,16 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "nf[\"nested.t\"][0]"
+    "nf[\"lightcurve.time\"][0]"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Here the `[0]` is telling our nested frame to access the values of the series `nf[\"nested.t\"]` where the id = 0. If we try `nf[\"nested.t\"][0][0]` we again match id = 0 and return the same frame. So to access a single element within the series, we can use its location:"
+    "Here the `[0]` is telling our nested frame to access the values of the series `nf[\"lightcurve.time\"]` where the id = 0. If we try `nf[\"lightcurve.time\"][0][0]` we again match id = 0 and return the same frame. \n",
+    "\n",
+    "To access a single element within the series, we need to use its location:"
    ]
   },
   {
@@ -172,7 +184,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "nf[\"nested.t\"][0].iloc[0]"
+    "nf[\"lightcurve.time\"][0].iloc[0]"
    ]
   },
   {
@@ -220,7 +232,7 @@
    "outputs": [],
    "source": [
     "# Normal queries work as expected, rejecting rows from the dataframe that don't meet the criteria\n",
-    "nf.query(\"a > 0.2\")"
+    "nf.query(\"ra > 11.2\")"
    ]
   },
   {
@@ -236,8 +248,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# Applies the query to \"nested\", filtering based on \"t >17\"\n",
-    "nf_g = nf.query(\"nested.t > 17.0\")\n",
+    "# Applies the query to \"nested\", filtering based on \"time > 60676.0\"\n",
+    "nf_g = nf.query(\"lightcurve.time > 60676.0\")\n",
     "nf_g"
    ]
   },
@@ -254,8 +266,8 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# All t <= 17.0 have been removed\n",
-    "nf_g.loc[0][\"nested\"]"
+    "# All t <= 60676.0 have been removed\n",
+    "nf_g.loc[0][\"lightcurve\"]"
    ]
   },
   {
@@ -284,7 +296,7 @@
     "\n",
     "# use hierarchical column names to access the flux column\n",
     "# passed as an array to np.mean\n",
-    "nf.reduce(np.mean, \"nested.flux\")"
+    "nf.reduce(np.mean, \"lightcurve.brightness\")"
    ]
   },
   {
@@ -308,7 +320,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Applying some inputs via reduce, we see how it sends inputs to a given function."
+    "Applying some inputs via reduce, we see how it sends inputs to a given function.  The output frame `nf_inputs` consists of two columns containing the output of the “ra” column and the “lightcurve.time” column."
    ]
   },
   {
@@ -317,7 +329,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "nf_inputs = nf.reduce(show_inputs, \"a\", \"nested.band\")\n",
+    "nf_inputs = nf.reduce(show_inputs, \"ra\", \"lightcurve.time\")\n",
     "nf_inputs"
    ]
   },
@@ -333,7 +345,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "tdastro",
+   "display_name": "nested",
    "language": "python",
    "name": "python3"
   },