examples/01_demo_processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8e79ad85-1172-4002-9f52-97c1e85cde44",
   "metadata": {},
   "source": [
    "# Demonstration notebook for processing raw RINEX data\n",
    "In this Notebook, we process some example RINEX files to demonstrate gnssvod."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c06a8784-1265-4ef9-8973-9455df3edd2d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gnssvod as gv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a548c0a-44f2-4548-8bfb-fc9348b49fa7",
   "metadata": {},
   "source": [
    "## gv.preprocess()\n",
    "The main pre-processing function is preprocess(). This function  will do several things\n",
    "- It will read RINEX observation files as pandas data frames\n",
    "- It can aggregate the raw data to a lower temporal rate if specified.\n",
    "- It will by default download orbit and clock files for the corresponding days from the GSSC ESA server\n",
    "- From the orbit and clock files, it will calculate azimuth and elevation for each measurement\n",
    "- It can save each processed file as a netcdf file in the outputdir folder or return the results as a dictionary\n",
    "\n",
    "### specifying input files\n",
    "The function exclusively reads RINEX observation files. Such files typically end with the extension '.yyO' where yy is the last two digit of the year. The function can be used to process a single file, a group of files, or several groups of files corresponding to several receivers, as shown in the examples below. All of this is done by specifying a pattern as the first argument to the function.\n",
    "\n",
    "### specifying output destinations\n",
    "Results are saved to a NetCDF file when an output directory is specified and/or returned as a dictionary when \"outputresult=True\" is passed.\n",
    "\n",
    "Let's read a single file using the example data to begin with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "6c8e3815-c887-48d4-8fdb-ab81701c29ff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O  is read in 2.69 seconds.\n",
      "Processing 112382 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "This file does not exist: GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3\n",
      "Downloading: GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz: 0.98MB [00:01, 769kB/s]                             \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " | Download completed for GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3.gz\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 2.12 seconds\n",
      "This file does not exist: GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3\n",
      "Downloading: GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz: 0.98MB [00:03, 336kB/s]                             \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " | Download completed for GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3.gz\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 3.76 seconds\n",
      "This file does not exist: GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK\n",
      "Downloading: GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz: 4.54MB [00:02, 2.34MB/s]                            \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " | Download completed for GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK.gz\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 2.01 seconds\n",
      "This file does not exist: GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK\n",
      "Downloading: GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz: 4.55MB [00:02, 2.34MB/s]                            \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " | Download completed for GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK.gz\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 1.91 seconds\n",
      "SP3 interpolation is done in 59.36 seconds\n"
     ]
    }
   ],
   "source": [
    "pattern = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O'}\n",
    "result = gv.preprocess(pattern,outputresult=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3d9f478-a625-4dd7-8f5d-71786face67f",
   "metadata": {},
   "source": [
    "The default logs should indicate how many observations were read in the file. If this is the first time you run the script, it also shows some orbit files were downloaded.\n",
    "\n",
    "If you process very recent data (less than 3 days old), it could be that the orbit and clock files are not available on the ESA server yet and there would then be an error.\n",
    "\n",
    "The result returned by the function is a dictionary providing lists of Observation objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fc4fe50a-c999-48ef-a385-2e6efd785c56",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'Dav2_Twr': [<gnssvod.io.io.Observation at 0x7fd179bd26a0>]}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f652b71f-aec9-427e-8e48-4c39479d3f19",
   "metadata": {},
   "source": [
    "Since we processed one file, there is only one Observation object in the list. Let us access this first and unique item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a4cd0168-3c4e-4450-a082-35646c4a42d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<gnssvod.io.io.Observation at 0x7fd179bd26a0>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs = result['Dav2_Twr'][0]\n",
    "obs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f858f1f5-a2d4-47fe-831a-146a755864ab",
   "metadata": {},
   "source": [
    "Observation objects are custom classes introduced in the `gnsspy` package by Mustafa Serkan Işık and Volkan Özbey. A significant number of base functions in `gnssvod` are based on gnsspy.\n",
    "\n",
    "Observation objects contain the following properties\n",
    "- obs.filename          = the name of the source file\n",
    "- obs.epoch             = a datetime indicate the day at the start of the record\n",
    "- obs.observation       = a pandas data frame containing all measurements\n",
    "- obs.approx_position   = the approximate receiver position as provided in the RINEX file [X,Y,Z]\n",
    "- obs.receiver_type     = the receiver type if provided in the RINEX file\n",
    "- obs.antenna_type      = the antenna type if provided in the RINEX file\n",
    "- obs.interval          = the measurement frequency in seconds\n",
    "- obs.receiver_clock    = the receiver clock if provided in the RINEX file\n",
    "- obs.version           = the version of the RINEX file\n",
    "- obs.observation_types = the observation types reported as columns in obs.observation\n",
    "\n",
    "Let's just look at the data.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7903891f-7535-4a13-b306-71da222c78d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>C1</th>\n",
       "      <th>L1</th>\n",
       "      <th>D1</th>\n",
       "      <th>S1</th>\n",
       "      <th>C2</th>\n",
       "      <th>L2</th>\n",
       "      <th>D2</th>\n",
       "      <th>S2</th>\n",
       "      <th>C7</th>\n",
       "      <th>L7</th>\n",
       "      <th>D7</th>\n",
       "      <th>S7</th>\n",
       "      <th>Azimuth</th>\n",
       "      <th>Elevation</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Epoch</th>\n",
       "      <th>SV</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">2021-04-28 21:07:08</th>\n",
       "      <th>G01</th>\n",
       "      <td>2.300005e+07</td>\n",
       "      <td>1.208661e+08</td>\n",
       "      <td>-2545.050</td>\n",
       "      <td>47.0</td>\n",
       "      <td>2.300006e+07</td>\n",
       "      <td>9.418140e+07</td>\n",
       "      <td>-1983.223</td>\n",
       "      <td>44.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>132.372864</td>\n",
       "      <td>59.902810</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>G03</th>\n",
       "      <td>2.257959e+07</td>\n",
       "      <td>1.186567e+08</td>\n",
       "      <td>-528.921</td>\n",
       "      <td>47.0</td>\n",
       "      <td>2.257959e+07</td>\n",
       "      <td>9.245973e+07</td>\n",
       "      <td>-412.197</td>\n",
       "      <td>42.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-16.744176</td>\n",
       "      <td>79.170432</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>G04</th>\n",
       "      <td>2.373431e+07</td>\n",
       "      <td>1.247247e+08</td>\n",
       "      <td>1971.157</td>\n",
       "      <td>45.0</td>\n",
       "      <td>2.373431e+07</td>\n",
       "      <td>9.718807e+07</td>\n",
       "      <td>1535.983</td>\n",
       "      <td>42.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-169.822432</td>\n",
       "      <td>50.693499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>G09</th>\n",
       "      <td>2.620530e+07</td>\n",
       "      <td>1.377098e+08</td>\n",
       "      <td>3155.798</td>\n",
       "      <td>40.0</td>\n",
       "      <td>2.620531e+07</td>\n",
       "      <td>1.073063e+08</td>\n",
       "      <td>2459.033</td>\n",
       "      <td>40.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-150.328510</td>\n",
       "      <td>19.288991</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>G17</th>\n",
       "      <td>2.423057e+07</td>\n",
       "      <td>1.273325e+08</td>\n",
       "      <td>-403.684</td>\n",
       "      <td>49.0</td>\n",
       "      <td>2.423058e+07</td>\n",
       "      <td>9.922014e+07</td>\n",
       "      <td>-314.543</td>\n",
       "      <td>40.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-81.696037</td>\n",
       "      <td>43.789922</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">2021-04-28 22:07:07</th>\n",
       "      <th>C16</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.282536e+08</td>\n",
       "      <td>-1695.972</td>\n",
       "      <td>36.0</td>\n",
       "      <td>4.383370e+07</td>\n",
       "      <td>2.282536e+08</td>\n",
       "      <td>-1695.972</td>\n",
       "      <td>36.0</td>\n",
       "      <td>4.383372e+07</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-1311.197</td>\n",
       "      <td>31.0</td>\n",
       "      <td>41.172558</td>\n",
       "      <td>7.343129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C27</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.432542e+08</td>\n",
       "      <td>1694.596</td>\n",
       "      <td>44.0</td>\n",
       "      <td>2.751047e+07</td>\n",
       "      <td>1.432542e+08</td>\n",
       "      <td>1694.596</td>\n",
       "      <td>44.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-42.395746</td>\n",
       "      <td>20.938009</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C28</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.326840e+08</td>\n",
       "      <td>-1191.327</td>\n",
       "      <td>49.0</td>\n",
       "      <td>2.548056e+07</td>\n",
       "      <td>1.326840e+08</td>\n",
       "      <td>-1191.327</td>\n",
       "      <td>49.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-99.551780</td>\n",
       "      <td>49.140759</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S23</th>\n",
       "      <td>4.111082e+07</td>\n",
       "      <td>2.160388e+08</td>\n",
       "      <td>-516.361</td>\n",
       "      <td>45.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S36</th>\n",
       "      <td>4.078565e+07</td>\n",
       "      <td>2.143301e+08</td>\n",
       "      <td>-517.120</td>\n",
       "      <td>42.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>112382 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   C1            L1        D1    S1  \\\n",
       "Epoch               SV                                                \n",
       "2021-04-28 21:07:08 G01  2.300005e+07  1.208661e+08 -2545.050  47.0   \n",
       "                    G03  2.257959e+07  1.186567e+08  -528.921  47.0   \n",
       "                    G04  2.373431e+07  1.247247e+08  1971.157  45.0   \n",
       "                    G09  2.620530e+07  1.377098e+08  3155.798  40.0   \n",
       "                    G17  2.423057e+07  1.273325e+08  -403.684  49.0   \n",
       "...                               ...           ...       ...   ...   \n",
       "2021-04-28 22:07:07 C16           NaN  2.282536e+08 -1695.972  36.0   \n",
       "                    C27           NaN  1.432542e+08  1694.596  44.0   \n",
       "                    C28           NaN  1.326840e+08 -1191.327  49.0   \n",
       "                    S23  4.111082e+07  2.160388e+08  -516.361  45.0   \n",
       "                    S36  4.078565e+07  2.143301e+08  -517.120  42.0   \n",
       "\n",
       "                                   C2            L2        D2    S2  \\\n",
       "Epoch               SV                                                \n",
       "2021-04-28 21:07:08 G01  2.300006e+07  9.418140e+07 -1983.223  44.0   \n",
       "                    G03  2.257959e+07  9.245973e+07  -412.197  42.0   \n",
       "                    G04  2.373431e+07  9.718807e+07  1535.983  42.0   \n",
       "                    G09  2.620531e+07  1.073063e+08  2459.033  40.0   \n",
       "                    G17  2.423058e+07  9.922014e+07  -314.543  40.0   \n",
       "...                               ...           ...       ...   ...   \n",
       "2021-04-28 22:07:07 C16  4.383370e+07  2.282536e+08 -1695.972  36.0   \n",
       "                    C27  2.751047e+07  1.432542e+08  1694.596  44.0   \n",
       "                    C28  2.548056e+07  1.326840e+08 -1191.327  49.0   \n",
       "                    S23           NaN           NaN       NaN   NaN   \n",
       "                    S36           NaN           NaN       NaN   NaN   \n",
       "\n",
       "                                   C7  L7        D7    S7     Azimuth  \\\n",
       "Epoch               SV                                                  \n",
       "2021-04-28 21:07:08 G01           NaN NaN       NaN   NaN  132.372864   \n",
       "                    G03           NaN NaN       NaN   NaN  -16.744176   \n",
       "                    G04           NaN NaN       NaN   NaN -169.822432   \n",
       "                    G09           NaN NaN       NaN   NaN -150.328510   \n",
       "                    G17           NaN NaN       NaN   NaN  -81.696037   \n",
       "...                               ...  ..       ...   ...         ...   \n",
       "2021-04-28 22:07:07 C16  4.383372e+07 NaN -1311.197  31.0   41.172558   \n",
       "                    C27           NaN NaN       NaN   NaN  -42.395746   \n",
       "                    C28           NaN NaN       NaN   NaN  -99.551780   \n",
       "                    S23           NaN NaN       NaN   NaN         NaN   \n",
       "                    S36           NaN NaN       NaN   NaN         NaN   \n",
       "\n",
       "                         Elevation  \n",
       "Epoch               SV              \n",
       "2021-04-28 21:07:08 G01  59.902810  \n",
       "                    G03  79.170432  \n",
       "                    G04  50.693499  \n",
       "                    G09  19.288991  \n",
       "                    G17  43.789922  \n",
       "...                            ...  \n",
       "2021-04-28 22:07:07 C16   7.343129  \n",
       "                    C27  20.938009  \n",
       "                    C28  49.140759  \n",
       "                    S23        NaN  \n",
       "                    S36        NaN  \n",
       "\n",
       "[112382 rows x 14 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "obs.observation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76f816b1-6353-48af-9aa5-5e621720a4fe",
   "metadata": {},
   "source": [
    "The pandas data frame has a MultIndex that contains both Epoch and SV as indices. The columns correspond to:\n",
    "- C# = Pseudorange from the receiver to the satellite, in meters\n",
    "- L# = Carrier phase, in cycles\n",
    "- D# = Doppler, in Hz\n",
    "- S# = Carrier to noise density C/N$_0$, in dB (receiver-dependent)\n",
    "\n",
    "And the numbers (S1, S2, etc. ) indicate the corresponding GNSS frequency\n",
    "\n",
    "The azimuth and elevation of the satellite with respect to the receiver are expressed in degrees. Computation speed for the azimuth and elevation can vary according to your hardware. Most of the time is spent interpolating the orbit parameters to the time stamps of each measurement. This is why it is sometimes useful to aggregate high frequency data (here one measurement per second) to for instance one measurement each 15 seconds.\n",
    "\n",
    "### resampling\n",
    "\n",
    "We can pass \"interval='15S'\" to resample the data during the preprocessing. The returned data will be smaller and the calculation of the azimuths and elevations (reported as \"SP3 interpolation\") will be faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d98edc38-4750-480b-9098-d7eb27b9a3df",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O  is read in 2.63 seconds.\n",
      "Processing 112382 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 1.32 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 1.22 seconds\n",
      "SP3 interpolation is done in 5.20 seconds\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th>C1</th>\n",
       "      <th>L1</th>\n",
       "      <th>D1</th>\n",
       "      <th>S1</th>\n",
       "      <th>C2</th>\n",
       "      <th>L2</th>\n",
       "      <th>D2</th>\n",
       "      <th>S2</th>\n",
       "      <th>C7</th>\n",
       "      <th>L7</th>\n",
       "      <th>D7</th>\n",
       "      <th>S7</th>\n",
       "      <th>Azimuth</th>\n",
       "      <th>Elevation</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Epoch</th>\n",
       "      <th>SV</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">2021-04-28 21:07:00</th>\n",
       "      <th>C06</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.243250e+08</td>\n",
       "      <td>-1539.026429</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>4.307927e+07</td>\n",
       "      <td>2.243250e+08</td>\n",
       "      <td>-1539.026429</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>4.307927e+07</td>\n",
       "      <td>1.734648e+08</td>\n",
       "      <td>-1189.841571</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>36.614495</td>\n",
       "      <td>10.132689</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C09</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.125070e+08</td>\n",
       "      <td>-1076.832857</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>4.080972e+07</td>\n",
       "      <td>2.125070e+08</td>\n",
       "      <td>-1076.832857</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>4.080973e+07</td>\n",
       "      <td>1.643239e+08</td>\n",
       "      <td>-832.746571</td>\n",
       "      <td>36.000000</td>\n",
       "      <td>49.034472</td>\n",
       "      <td>32.742503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C11</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.352152e+08</td>\n",
       "      <td>-3555.085714</td>\n",
       "      <td>43.428571</td>\n",
       "      <td>2.596665e+07</td>\n",
       "      <td>1.352152e+08</td>\n",
       "      <td>-3555.085714</td>\n",
       "      <td>43.428571</td>\n",
       "      <td>2.596666e+07</td>\n",
       "      <td>1.045570e+08</td>\n",
       "      <td>-2748.967571</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>177.234549</td>\n",
       "      <td>35.079888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C14</th>\n",
       "      <td>NaN</td>\n",
       "      <td>1.240611e+08</td>\n",
       "      <td>32.760429</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>2.382462e+07</td>\n",
       "      <td>1.240611e+08</td>\n",
       "      <td>32.760429</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>2.382462e+07</td>\n",
       "      <td>9.593200e+07</td>\n",
       "      <td>25.282429</td>\n",
       "      <td>42.285714</td>\n",
       "      <td>-96.373353</td>\n",
       "      <td>76.785189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>C16</th>\n",
       "      <td>NaN</td>\n",
       "      <td>2.222891e+08</td>\n",
       "      <td>-1595.855857</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>4.268828e+07</td>\n",
       "      <td>2.222891e+08</td>\n",
       "      <td>-1595.855857</td>\n",
       "      <td>38.000000</td>\n",
       "      <td>4.268830e+07</td>\n",
       "      <td>1.718881e+08</td>\n",
       "      <td>-1234.117000</td>\n",
       "      <td>33.000000</td>\n",
       "      <td>38.266603</td>\n",
       "      <td>15.249211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th rowspan=\"5\" valign=\"top\">2021-04-28 22:07:00</th>\n",
       "      <th>R18</th>\n",
       "      <td>2.326705e+07</td>\n",
       "      <td>1.242018e+08</td>\n",
       "      <td>-3848.505125</td>\n",
       "      <td>46.375000</td>\n",
       "      <td>2.326705e+07</td>\n",
       "      <td>9.660093e+07</td>\n",
       "      <td>-2993.250750</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-167.160718</td>\n",
       "      <td>45.315939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R19</th>\n",
       "      <td>2.237872e+07</td>\n",
       "      <td>1.197111e+08</td>\n",
       "      <td>-52.467125</td>\n",
       "      <td>37.375000</td>\n",
       "      <td>2.237872e+07</td>\n",
       "      <td>9.310873e+07</td>\n",
       "      <td>-40.688500</td>\n",
       "      <td>38.875000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-74.711561</td>\n",
       "      <td>63.385367</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>R20</th>\n",
       "      <td>2.596147e+07</td>\n",
       "      <td>1.388277e+08</td>\n",
       "      <td>3565.564875</td>\n",
       "      <td>37.875000</td>\n",
       "      <td>2.596148e+07</td>\n",
       "      <td>1.079770e+08</td>\n",
       "      <td>2773.281125</td>\n",
       "      <td>34.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>-26.146660</td>\n",
       "      <td>13.545577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S23</th>\n",
       "      <td>4.111048e+07</td>\n",
       "      <td>2.160370e+08</td>\n",
       "      <td>-515.803000</td>\n",
       "      <td>44.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>S36</th>\n",
       "      <td>4.078531e+07</td>\n",
       "      <td>2.143283e+08</td>\n",
       "      <td>-516.305375</td>\n",
       "      <td>41.750000</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>7550 rows × 14 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   C1            L1           D1         S1  \\\n",
       "Epoch               SV                                                        \n",
       "2021-04-28 21:07:00 C06           NaN  2.243250e+08 -1539.026429  38.000000   \n",
       "                    C09           NaN  2.125070e+08 -1076.832857  41.000000   \n",
       "                    C11           NaN  1.352152e+08 -3555.085714  43.428571   \n",
       "                    C14           NaN  1.240611e+08    32.760429  45.000000   \n",
       "                    C16           NaN  2.222891e+08 -1595.855857  38.000000   \n",
       "...                               ...           ...          ...        ...   \n",
       "2021-04-28 22:07:00 R18  2.326705e+07  1.242018e+08 -3848.505125  46.375000   \n",
       "                    R19  2.237872e+07  1.197111e+08   -52.467125  37.375000   \n",
       "                    R20  2.596147e+07  1.388277e+08  3565.564875  37.875000   \n",
       "                    S23  4.111048e+07  2.160370e+08  -515.803000  44.750000   \n",
       "                    S36  4.078531e+07  2.143283e+08  -516.305375  41.750000   \n",
       "\n",
       "                                   C2            L2           D2         S2  \\\n",
       "Epoch               SV                                                        \n",
       "2021-04-28 21:07:00 C06  4.307927e+07  2.243250e+08 -1539.026429  38.000000   \n",
       "                    C09  4.080972e+07  2.125070e+08 -1076.832857  41.000000   \n",
       "                    C11  2.596665e+07  1.352152e+08 -3555.085714  43.428571   \n",
       "                    C14  2.382462e+07  1.240611e+08    32.760429  45.000000   \n",
       "                    C16  4.268828e+07  2.222891e+08 -1595.855857  38.000000   \n",
       "...                               ...           ...          ...        ...   \n",
       "2021-04-28 22:07:00 R18  2.326705e+07  9.660093e+07 -2993.250750  42.000000   \n",
       "                    R19  2.237872e+07  9.310873e+07   -40.688500  38.875000   \n",
       "                    R20  2.596148e+07  1.079770e+08  2773.281125  34.750000   \n",
       "                    S23           NaN           NaN          NaN        NaN   \n",
       "                    S36           NaN           NaN          NaN        NaN   \n",
       "\n",
       "                                   C7            L7           D7         S7  \\\n",
       "Epoch               SV                                                        \n",
       "2021-04-28 21:07:00 C06  4.307927e+07  1.734648e+08 -1189.841571  31.000000   \n",
       "                    C09  4.080973e+07  1.643239e+08  -832.746571  36.000000   \n",
       "                    C11  2.596666e+07  1.045570e+08 -2748.967571  41.000000   \n",
       "                    C14  2.382462e+07  9.593200e+07    25.282429  42.285714   \n",
       "                    C16  4.268830e+07  1.718881e+08 -1234.117000  33.000000   \n",
       "...                               ...           ...          ...        ...   \n",
       "2021-04-28 22:07:00 R18           NaN           NaN          NaN        NaN   \n",
       "                    R19           NaN           NaN          NaN        NaN   \n",
       "                    R20           NaN           NaN          NaN        NaN   \n",
       "                    S23           NaN           NaN          NaN        NaN   \n",
       "                    S36           NaN           NaN          NaN        NaN   \n",
       "\n",
       "                            Azimuth  Elevation  \n",
       "Epoch               SV                          \n",
       "2021-04-28 21:07:00 C06   36.614495  10.132689  \n",
       "                    C09   49.034472  32.742503  \n",
       "                    C11  177.234549  35.079888  \n",
       "                    C14  -96.373353  76.785189  \n",
       "                    C16   38.266603  15.249211  \n",
       "...                             ...        ...  \n",
       "2021-04-28 22:07:00 R18 -167.160718  45.315939  \n",
       "                    R19  -74.711561  63.385367  \n",
       "                    R20  -26.146660  13.545577  \n",
       "                    S23         NaN        NaN  \n",
       "                    S36         NaN        NaN  \n",
       "\n",
       "[7550 rows x 14 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pattern = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O'}\n",
    "result = gv.preprocess(pattern,interval='15S',outputresult=True)\n",
    "# and show data frame\n",
    "result['Dav2_Twr'][0].observation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bb50205-b28e-4d3f-a037-f9e9d8159c67",
   "metadata": {},
   "source": [
    "Orbit and clock files are not downloaded again if they already exist. There are now less rows in the data frame."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d478c7b1-74ce-4da7-89b7-d8690011c37c",
   "metadata": {},
   "source": [
    "## Batch processing\n",
    "We now use the preprocessing function to process many files and save the outputs as NetCDF files (instead of returning as objects). If we were to process several hundreds of files, the system would likely not have sufficient memory to hold all of the outputs, so it makes sense to processed data as a NetCDF file.\n",
    "\n",
    "### Specifying several groups of files\n",
    "Instead of specifying just one file, we use the dictionary to specify a pattern. All files matching the pattern will be processed. We can process several groups files by specifying different matching patterns (see below).\n",
    "\n",
    "### Specifying where to save data\n",
    "Same as for specifying the inputs, we use a dictionary to indicate where to save data. The function will create the destination folder if it does not exist.\n",
    "\n",
    "### Specifying a list of variables to save\n",
    "For calculating GNSS-VOD, we only need the \"S\" variables. We can reduce the size of the saved NetCDF files by discarding the other variables, this is done with the 'keepvars' argument, which will only keep the variables present in the passed list. This argument supports UNIX-style pattern matching (e.g. 'S*' will match all variables starting with 'S')\n",
    "\n",
    "### Compression\n",
    "Unless `compress=False` is passed as argument, `gv.preprocess()` will compress all S* variables, as well as Azimuth and Elevation when saving to NetCDF. These variables are encoded as Int16 with a scale factor of 0.1. The decoding is automatically applied when reading the data with xarray."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0e7e75d9-c77a-4b5c-9752-2317ac852696",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/huvincen/Work/Projects/GNSS/gnssvod/gnssvod/io/preprocess.py:245: UserWarning: Could not find any files matching the pattern data_RINEX2.11/Dav2_Twr/nc/*.nc\n",
      "  warnings.warn(f\"Could not find any files matching the pattern {search_pattern}\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290006.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290006.21O  is read in 2.42 seconds.\n",
      "Processing 109702 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.27 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 1.77 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 2.06 seconds\n",
      "SP3 interpolation is done in 5.37 seconds\n",
      "Saved 7370 individual observations in Reach_Dav2_Twr-raw_202104290006.nc\n",
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290106.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290106.21O  is read in 2.82 seconds.\n",
      "Processing 117216 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 7871 individual observations in Reach_Dav2_Twr-raw_202104290106.nc\n",
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290206.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104290206.21O  is read in 2.67 seconds.\n",
      "Processing 108484 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 7328 individual observations in Reach_Dav2_Twr-raw_202104290206.nc\n",
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282106.21O  is read in 2.67 seconds.\n",
      "Processing 112382 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 1.70 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 1.95 seconds\n",
      "SP3 interpolation is done in 5.45 seconds\n",
      "Saved 7550 individual observations in Reach_Dav2_Twr-raw_202104282106.nc\n",
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282206.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282206.21O  is read in 2.52 seconds.\n",
      "Processing 112113 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 7533 individual observations in Reach_Dav2_Twr-raw_202104282206.nc\n",
      "data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282306.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav2_Twr/rinex/Reach_Dav2_Twr-raw_202104282306.21O  is read in 2.57 seconds.\n",
      "Processing 115429 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 7756 individual observations in Reach_Dav2_Twr-raw_202104282306.nc\n",
      "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290006.21O exists | Reading...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/huvincen/Work/Projects/GNSS/gnssvod/gnssvod/io/preprocess.py:245: UserWarning: Could not find any files matching the pattern data_RINEX2.11/Dav1_Grnd/nc/*.nc\n",
      "  warnings.warn(f\"Could not find any files matching the pattern {search_pattern}\")\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Observation file  data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290006.21O  is read in 2.44 seconds.\n",
      "Processing 110793 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.43 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 1.74 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 1.72 seconds\n",
      "SP3 interpolation is done in 5.25 seconds\n",
      "Saved 7493 individual observations in Reach_Dav1_Grnd-raw_202104290006.nc\n",
      "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290106.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290106.21O  is read in 2.44 seconds.\n",
      "Processing 109171 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 7411 individual observations in Reach_Dav1_Grnd-raw_202104290106.nc\n",
      "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290206.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104290206.21O  is read in 2.47 seconds.\n",
      "Processing 102591 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 6959 individual observations in Reach_Dav1_Grnd-raw_202104290206.nc\n",
      "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282106.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282106.21O  is read in 2.65 seconds.\n",
      "Processing 111162 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_05M_ORB.SP3 file is read in 0.28 seconds\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211180000_01D_30S_CLK.CLK file is read in 1.74 seconds\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK exists | Reading...\n",
      "GFZ0MGXRAP_20211190000_01D_30S_CLK.CLK file is read in 1.68 seconds\n",
      "SP3 interpolation is done in 5.22 seconds\n",
      "Saved 7481 individual observations in Reach_Dav1_Grnd-raw_202104282106.nc\n",
      "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282206.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282206.21O  is read in 2.43 seconds.\n",
      "Processing 107018 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 7231 individual observations in Reach_Dav1_Grnd-raw_202104282206.nc\n",
      "data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282306.21O exists | Reading...\n",
      "Observation file  data_RINEX2.11/Dav1_Grnd/rinex/Reach_Dav1_Grnd-raw_202104282306.21O  is read in 2.52 seconds.\n",
      "Processing 113394 individual observations\n",
      "Calculating Azimuth and Elevation\n",
      "Saved 7670 individual observations in Reach_Dav1_Grnd-raw_202104282306.nc\n"
     ]
    }
   ],
   "source": [
    "# use gnssvod to batch process the observation RINEX files \n",
    "# (files with extension .yyO for each station)\n",
    "# pattern = {'choice_of_name_for_station1':'pattern to match (UNIX-style)',\n",
    "#            'choice_of_name_for_station2':'pattern to match (UNIX-style)',\n",
    "#             ...}\n",
    "#\n",
    "pattern = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/rinex/*.*O',\n",
    "          'Dav1_Grnd':'data_RINEX2.11/Dav1_Grnd/rinex/*.*O'}\n",
    "outputdir = {'Dav2_Twr':'data_RINEX2.11/Dav2_Twr/nc/',\n",
    "            'Dav1_Grnd':'data_RINEX2.11/Dav1_Grnd/nc/'}\n",
    "# what variables should be kept\n",
    "keepvars = ['S?','S??']\n",
    "\n",
    "gv.preprocess(pattern,interval='15S',keepvars=keepvars,outputdir=outputdir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea7754bc-2319-4102-ae32-4c195724a89d",
   "metadata": {},
   "source": [
    "### Skipping existing files by default\n",
    "The preprocess function will scan the destination folder for existing NetCDF files. If some files are found that have already been processed, these files will be skipped unless overwrite=True has been passed.\n",
    "\n",
    "Here because the destination folder was empty, a user warning appears in the log above but can be ignored (\"Could not find any files matching the pattern data_RINEX2.11/Dav2_Twr/nc/*.nc\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:.conda-gnss_2022]",
   "language": "python",
   "name": "conda-env-.conda-gnss_2022-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}