2 Basic xarray - subsetting data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Subsetting a dataset by time and space (Slicing and Dicing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reload the library and the dataset from the previous notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = xarray.open_dataset('http://dapds00.nci.org.au/thredds/dodsC/rr3/CMIP5/output1/CSIRO-BOM/ACCESS1-3/historical/mon/atmos/Amon/r1i1p1/latest/tas/tas_Amon_ACCESS1-3_historical_r1i1p1_185001-200512.nc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select just the data variable `tas` and save it to another variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 1872, lat: 145, lon: 192)>\n",
       "[52116480 values with dtype=float32]\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1850-01-16T12:00:00 ... 2005-12-16T12:00:00\n",
       "  * lat      (lat) float64 -90.0 -88.75 -87.5 -86.25 ... 86.25 87.5 88.75 90.0\n",
       "  * lon      (lon) float64 0.0 1.875 3.75 5.625 7.5 ... 352.5 354.4 356.2 358.1\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas = ds.tas\n",
    "tas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "xarray builds on top of numpy, and stores its data internally as numpy arrays. It supports many numpy operations, so it is possible to find out the shape of the underlying data, and use numpy style indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1872, 145, 192)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (lat: 145, lon: 192)>\n",
       "array([[240.83618, 240.83618, 240.83618, ..., 240.83191, 240.83191, 240.83191],\n",
       "       [242.67894, 242.65828, 242.63867, ..., 242.72214, 242.71056, 242.69565],\n",
       "       [243.58124, 243.52393, 243.46175, ..., 243.73363, 243.69022, 243.6365 ],\n",
       "       ...,\n",
       "       [239.76013, 239.84523, 239.93364, ..., 239.52486, 239.60957, 239.68538],\n",
       "       [239.02599, 239.03561, 239.04205, ..., 239.04428, 239.03633, 239.02672],\n",
       "       [238.45506, 238.45506, 238.45506, ..., 238.45506, 238.45506, 238.45506]],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "    time     datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -90.0 -88.75 -87.5 -86.25 ... 86.25 87.5 88.75 90.0\n",
       "  * lon      (lon) float64 0.0 1.875 3.75 5.625 7.5 ... 352.5 354.4 356.2 358.1\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas[0,...]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By selecting just the first time index it has created a DataArray with no time dimension, but time is still a coordinate not associated with any variable, as indicated by no longer having `*` beside it. It now only has one value: the value of the first time index. The index selection above is equivalent to using `isel` like so"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (lat: 145, lon: 192)>\n",
       "array([[240.83618, 240.83618, 240.83618, ..., 240.83191, 240.83191, 240.83191],\n",
       "       [242.67894, 242.65828, 242.63867, ..., 242.72214, 242.71056, 242.69565],\n",
       "       [243.58124, 243.52393, 243.46175, ..., 243.73363, 243.69022, 243.6365 ],\n",
       "       ...,\n",
       "       [239.76013, 239.84523, 239.93364, ..., 239.52486, 239.60957, 239.68538],\n",
       "       [239.02599, 239.03561, 239.04205, ..., 239.04428, 239.03633, 239.02672],\n",
       "       [238.45506, 238.45506, 238.45506, ..., 238.45506, 238.45506, 238.45506]],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "    time     datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -90.0 -88.75 -87.5 -86.25 ... 86.25 87.5 88.75 90.0\n",
       "  * lon      (lon) float64 0.0 1.875 3.75 5.625 7.5 ... 352.5 354.4 356.2 358.1\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.isel(time=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way is more compact, and one more descriptive, but they have the same result."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The power of xarray comes with the close association of data with coordinates. So it is possible to use the equivalent `.sel` operator but with coordinate values. For example, to select an area that includes the Indian Ocean and Australia use `slice` to indicate the range of latitude and longitude values required and pass as key/value pairs to `sel`. `slice` will include coordinate values inferior **or equal** to the upper bound, not like `range` in basic Python that excludes the upper bound."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 1872, lat: 85, lon: 75)>\n",
       "[11934000 values with dtype=float32]\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1850-01-16T12:00:00 ... 2005-12-16T12:00:00\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(lon=slice(20,160),lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Operators can be chained, so multiple operations can be peformed sequentially. For example, to select the above areas and the first time index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (lat: 85, lon: 75)>\n",
       "array([[244.32095, 244.06064, 243.80359, ..., 251.85716, 253.08392, 254.23578],\n",
       "       [243.11086, 242.9945 , 242.96965, ..., 250.56412, 252.03651, 253.29895],\n",
       "       [242.59998, 242.75227, 242.95288, ..., 251.33804, 252.93666, 254.82977],\n",
       "       ...,\n",
       "       [280.82947, 281.203  , 281.8401 , ..., 298.99326, 299.03836, 299.07532],\n",
       "       [280.9224 , 281.0952 , 281.43793, ..., 297.9115 , 298.03323, 298.08383],\n",
       "       [280.7901 , 281.48508, 281.8507 , ..., 296.3739 , 296.66852, 296.84888]],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "    time     datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.isel(time=0).sel(lon=slice(20,160), lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case it is more convenient to use `isel` to select the time, rather than specifying a date, but this would also work"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 1, lat: 85, lon: 75)>\n",
       "array([[[244.32095, 244.06064, ..., 253.08392, 254.23578],\n",
       "        [243.11086, 242.9945 , ..., 252.03651, 253.29895],\n",
       "        ...,\n",
       "        [280.9224 , 281.0952 , ..., 298.03323, 298.08383],\n",
       "        [280.7901 , 281.48508, ..., 296.66852, 296.84888]]], dtype=float32)\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1850-01-16T12:00:00\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(time='1850-01-16', lon=slice(20,160), lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note when using `sel` to select only 1 time index, the resulting array still has a time dimension of length 1. Depending on what you want to do with the array afterwards, it might be better to still have the time dimension or not. \n",
    "\n",
    "It is also possible to use `slice` for the `time` dimension. To select Mar to November of 1871:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 9, lat: 85, lon: 75)>\n",
       "array([[[232.72298, 232.41539, ..., 240.3316 , 241.3103 ],\n",
       "        [230.3484 , 230.09096, ..., 241.09224, 242.43002],\n",
       "        ...,\n",
       "        [287.8961 , 287.78564, ..., 296.20505, 296.41022],\n",
       "        [288.0455 , 288.2771 , ..., 294.80002, 294.9939 ]],\n",
       "\n",
       "       [[230.42459, 229.92857, ..., 240.06776, 241.25368],\n",
       "        [227.20078, 226.75165, ..., 239.50015, 241.08636],\n",
       "        ...,\n",
       "        [293.6292 , 293.43594, ..., 297.73218, 297.78775],\n",
       "        [293.78354, 294.1199 , ..., 296.54602, 296.6775 ]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[234.84286, 234.5873 , ..., 242.15875, 243.17744],\n",
       "        [233.36253, 233.07495, ..., 242.26463, 243.54987],\n",
       "        ...,\n",
       "        [290.4821 , 290.44135, ..., 300.78342, 300.67822],\n",
       "        [290.74014, 291.0542 , ..., 300.5218 , 300.38864]],\n",
       "\n",
       "       [[237.83063, 237.62698, ..., 248.45111, 249.83937],\n",
       "        [236.2484 , 235.9952 , ..., 247.79037, 249.23076],\n",
       "        ...,\n",
       "        [286.57666, 286.7646 , ..., 299.87238, 299.7646 ],\n",
       "        [286.386  , 287.12302, ..., 299.55652, 299.45605]]], dtype=float32)\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1871-03-16T12:00:00 1871-04-16 ... 1871-11-16\n",
       "  * lat      (lat) float64 -80.0 -78.75 -77.5 -76.25 ... 21.25 22.5 23.75 25.0\n",
       "  * lon      (lon) float64 20.62 22.5 24.38 26.25 ... 153.8 155.6 157.5 159.4\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(time=slice('1871-03','1871-11'), lon=slice(20,160), lat=slice(-80,25))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `slice` operator selects values between an upper and lower bound. If a single coordinate value is required when using `sel` it must either correspond to an *exact* value in the coordinate array, or the `method` argument specified to tell xarray how to choose a value. For example, to select out just values in the cell closest to Brisbane"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<xarray.DataArray 'tas' (time: 1872)>\n",
       "array([296.4649 , 296.54944, 296.43167, ..., 293.6983 , 294.73325, 296.40576],\n",
       "      dtype=float32)\n",
       "Coordinates:\n",
       "  * time     (time) datetime64[ns] 1850-01-16T12:00:00 ... 2005-12-16T12:00:00\n",
       "    lat      float64 -27.5\n",
       "    lon      float64 153.8\n",
       "    height   float64 ...\n",
       "Attributes:\n",
       "    standard_name:     air_temperature\n",
       "    long_name:         Near-Surface Air Temperature\n",
       "    units:             K\n",
       "    cell_methods:      time: mean\n",
       "    cell_measures:     area: areacella\n",
       "    history:           2012-02-05T23:49:51Z altered by CMOR: Treated scalar d...\n",
       "    associated_files:  baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation..."
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tas.sel(lat=-27.47, lon=153.03, method='nearest')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}