examples/02_demo_gathering.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a267e0f9-b4b8-45ab-9d70-d7045571d9e6",
   "metadata": {},
   "source": [
    "# Merging processed data\n",
    "This notebook relies on the data from the previous notebook (but there is no need to run the previous notebook for this one to work however)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2fedbcaf-dde3-436d-bf2d-5cc0b24129a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gnssvod as gv\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1da3daa-f534-4957-8d4c-09d7ec50d67b",
   "metadata": {},
   "source": [
    "## Merge\n",
    "In the previous notebook, we processed raw RINEX observation files individually for each receiver and saved the results in corresponding NetCDF files.\n",
    "\n",
    "In the case of a GNSS-VOD set up, receivers are analysed as pairs. One receiver lies above the forest canopy and provides a clear-sky reference, and the other one lies below the canopy and measures the forest attenuation.\n",
    "\n",
    "Here we merge the data from these two receivers before making any plots. We also save the merged data in chunks that are always the same (for example we save them in daily chunks). This makes it easier to manipulate data and avoids relying on the temporal chunks with which data was initially logged (here data was logged in hourly log files that span from xx:07 too xx+1:06).\n",
    "\n",
    "### gv.gather_stations()\n",
    "This function will do several things\n",
    "- It will read processed observation files that were saved in NetCDF format (output of \"preprocess\").\n",
    "- It will combine data from the various receivers/stations according to user-specified pairing rules.\n",
    "- It will only process data belonging to the requested time interval.\n",
    "- It will return and/or save paired data in temporal chunks specified by the time interval.\n",
    "\n",
    "#### Specifying input files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fa2e95bb-d5a9-4887-9206-aec3223f8fb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# first let's indicate where to find the data for each receiver\n",
    "pattern={'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/nc/*.nc',\n",
    "         'Dav1_Grnd':'data_RINEX2.11/Dav1_Grnd/nc/*.nc'}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "064dc425-b818-4ec8-b893-e5387ea2c0fa",
   "metadata": {},
   "source": [
    "#### Specifying time interval\n",
    "Then we need to define the temporal interval and the temporal chunks we will want for the output data\n",
    "                                                                             \n",
    "Here we decide to process all data from '28-04-2021' to '29-04-2021', meaning 2 days, starting at '28-04-2021'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "df7bcab7-266b-4e47-b016-bb66f5b6a87c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "IntervalIndex([[2021-04-28, 2021-04-29), [2021-04-29, 2021-04-30)], dtype='interval[datetime64[ns], left]')"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "startday = start=pd.to_datetime('28-04-2021',format='%d-%m-%Y')\n",
    "timeintervals=pd.interval_range(start=startday, periods=2, freq='D', closed='left')\n",
    "timeintervals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4716a18c-aec0-4f30-b143-3ccd6cf73218",
   "metadata": {},
   "source": [
    "Using the timeintervals above will save/return the results in chunks of 1 day. If we wanted the results in hourly chunks, we could have written instead:\n",
    "\n",
    "`timeintervals=pd.interval_range(start=startday, periods=48, freq='H', closed='left')`\n",
    "\n",
    "Now the only thing left is to define how to combine the stations, using the same dictionary keys as in 'pattern'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "56232ec2-f8e8-4421-ad97-6400919e88a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing Dav\n",
      "Listing the files matching with the interval\n",
      "Found 6 files for Dav2_Twr\n",
      "Reading\n",
      "Found 6 files for Dav1_Grnd\n",
      "Reading\n",
      "Concatenating\n"
     ]
    }
   ],
   "source": [
    "# define how to make pairs, always give reference station first, matching the dictionary keys of 'pattern'\n",
    "pairings={'Dav':('Dav2_Twr','Dav1_Grnd')}\n",
    "\n",
    "# run function\n",
    "out = gv.gather_stations(pattern,pairings,timeintervals)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ffc79c1-19e9-44d3-9960-d4b71b8bf251",
   "metadata": {},
   "source": [
    "The result is of the form\n",
    "\n",
    "out = dict(key:list(\n",
    "<br>&emsp;&emsp;tuple(pd.Interval,pd.DataFrame)),\n",
    "<br>&emsp;&emsp;tuple(pd.Interval,pd.DataFrame)),\n",
    "<br>&emsp;&emsp;tuple(pd.Interval,pd.DataFrame))\n",
    "<br>)\n",
    "\n",
    "In our case, something like:\n",
    "\n",
    "out = dict('Dav': \\[\n",
    "<br>&emsp;&emsp;(Interval('2021-04-28', '2021-04-29', closed='left'), dataframe),\n",
    "<br>&emsp;&emsp;(Interval('2021-04-29', '2021-04-30', closed='left'), dataframe)\n",
    "<br>\\])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3fad3763-17c3-402d-a1a0-505388754fee",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>S1</th>\n",
       "      <th>S2</th>\n",
       "      <th>S7</th>\n",
       "      <th>Azimuth</th>\n",
       "      <th>Elevation</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Station</th>\n",
       "      <th>Epoch</th>\n",
       "      <th>SV</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Dav2_Twr</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">2021-04-28 21:07:00</th>\n",
       "      <th>C06</th>\n",
       "      <td>38.000000</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>36.600002</td>\n",
       "      <td>10.100000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C09</th>\n",
       "      <td>41.000000</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>36.000000</td>\n",
       "      <td>49.000000</td>\n",
       "      <td>32.700001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C11</th>\n",
       "      <td>43.400002</td>\n",
       "      <td>43.400002</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>177.199997</td>\n",
       "      <td>35.100002</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C14</th>\n",
       "      <td>45.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>42.299999</td>\n",
       "      <td>-96.400002</td>\n",
       "      <td>76.800003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C16</th>\n",
       "      <td>38.000000</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>33.000000</td>\n",
       "      <td>38.299999</td>\n",
       "      <td>15.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">Dav1_Grnd</th>\n",
       "      <th rowspan=\"5\" valign=\"top\">2021-04-28 23:59:45</th>\n",
       "      <th>R20</th>\n",
       "      <td>39.000000</td>\n",
       "      <td>35.299999</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-81.099998</td>\n",
       "      <td>43.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R21</th>\n",
       "      <td>33.600002</td>\n",
       "      <td>34.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-21.200001</td>\n",
       "      <td>14.700000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S23</th>\n",
       "      <td>35.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S27</th>\n",
       "      <td>31.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S36</th>\n",
       "      <td>32.500000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>43172 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                          S1         S2         S7  \\\n",
       "Station   Epoch               SV                                     \n",
       "Dav2_Twr  2021-04-28 21:07:00 C06  38.000000  38.000000  31.000000   \n",
       "                              C09  41.000000  41.000000  36.000000   \n",
       "                              C11  43.400002  43.400002  41.000000   \n",
       "                              C14  45.000000  45.000000  42.299999   \n",
       "                              C16  38.000000  38.000000  33.000000   \n",
       "...                                      ...        ...        ...   \n",
       "Dav1_Grnd 2021-04-28 23:59:45 R20  39.000000  35.299999        NaN   \n",
       "                              R21  33.600002  34.000000        NaN   \n",
       "                              S23  35.000000        NaN        NaN   \n",
       "                              S27  31.000000        NaN        NaN   \n",
       "                              S36  32.500000        NaN        NaN   \n",
       "\n",
       "                                      Azimuth  Elevation  \n",
       "Station   Epoch               SV                          \n",
       "Dav2_Twr  2021-04-28 21:07:00 C06   36.600002  10.100000  \n",
       "                              C09   49.000000  32.700001  \n",
       "                              C11  177.199997  35.100002  \n",
       "                              C14  -96.400002  76.800003  \n",
       "                              C16   38.299999  15.200000  \n",
       "...                                       ...        ...  \n",
       "Dav1_Grnd 2021-04-28 23:59:45 R20  -81.099998  43.000000  \n",
       "                              R21  -21.200001  14.700000  \n",
       "                              S23         NaN        NaN  \n",
       "                              S27         NaN        NaN  \n",
       "                              S36         NaN        NaN  \n",
       "\n",
       "[43172 rows x 5 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out['Dav'][0][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f07f5f97-80f1-4f32-9358-2142a36fb755",
   "metadata": {},
   "source": [
    "#### Specifying output destination\n",
    "Instead of just returning the result as an output of the function, we can specify where to save it instead. Again it may also be useful to get rid of some variables that are not useful to reduce file size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "351245e9-b575-4433-a78f-fc3e08130672",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing Dav\n",
      "Listing the files matching with the interval\n",
      "Found 6 files for Dav2_Twr\n",
      "Reading\n",
      "Found 6 files for Dav1_Grnd\n",
      "Reading\n",
      "Concatenating\n",
      "Saving files for Dav in data_RINEX2.11/Dav_paired/\n",
      "Saved 43172 obs in Dav_20210428000000_20210429000000.nc\n",
      "Saved 46164 obs in Dav_20210429000000_20210430000000.nc\n"
     ]
    }
   ],
   "source": [
    "# define where to save output data, matching the dictionary keys in 'pairings'\n",
    "outputdir = {'Dav':'data_RINEX2.11/Dav_paired/'}\n",
    "# define which variables to keep\n",
    "keepvars = ['S1','S2','Azimuth','Elevation']\n",
    "\n",
    "# run function\n",
    "out = gv.gather_stations(pattern,pairings,timeintervals,keepvars=keepvars,outputdir=outputdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a14ccda-7cde-4ece-a465-be4114a08d6a",
   "metadata": {},
   "source": [
    "As we asked, the results have been saved as daily files (even though the input files are hourly files). The file names are generated based on the key of the 'pairing' argument (here 'Dav') and the specified time intervals."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:.conda-gnss_2022]",
   "language": "python",
   "name": "conda-env-.conda-gnss_2022-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}