diff --git a/validation/v1.0.0/horizontal_regridding/regrid_cdat_xarray.ipynb b/validation/v0.3.0/horizontal_regridding/regrid_cdat_xarray.ipynb similarity index 100% rename from validation/v1.0.0/horizontal_regridding/regrid_cdat_xarray.ipynb rename to validation/v0.3.0/horizontal_regridding/regrid_cdat_xarray.ipynb diff --git a/validation/v1.0.0/horizontal_regridding/regrid_cdat_xarray_xcdat.ipynb b/validation/v0.3.0/horizontal_regridding/regrid_cdat_xarray_xcdat.ipynb similarity index 100% rename from validation/v1.0.0/horizontal_regridding/regrid_cdat_xarray_xcdat.ipynb rename to validation/v0.3.0/horizontal_regridding/regrid_cdat_xarray_xcdat.ipynb diff --git a/validation/v1.0.0/horizontal_regridding/regrid_cdat_xcdat.ipynb b/validation/v0.3.0/horizontal_regridding/regrid_cdat_xcdat.ipynb similarity index 100% rename from validation/v1.0.0/horizontal_regridding/regrid_cdat_xcdat.ipynb rename to validation/v0.3.0/horizontal_regridding/regrid_cdat_xcdat.ipynb diff --git a/validation/v0.3.0/spatial_average/output_dtype_comparison.ipynb b/validation/v0.3.0/spatial_average/output_dtype_comparison.ipynb new file mode 100644 index 0000000..b3b146d --- /dev/null +++ b/validation/v0.3.0/spatial_average/output_dtype_comparison.ipynb @@ -0,0 +1,784 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# xCDAT vs. CDAT Spatial Averaging Output `dtype`\n", + "\n", + "Objective:\n", + "\n", + "* Figure out the root cause for the large floating point differences (absolute and relative), which might be related to the `dtype`.\n", + "\n", + "Questions:\n", + "* Is xCDAT or CDAT doing something incorrectly?\n", + "* Does CDAT cast the type of the data from `float32` to `float64`?\n", + " * If it does, when does it cast the type?" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Resources:\n", + "https://discourse.pangeo.io/t/variable-type-changing-when-using-xarray/1823/2\n", + "\n", + "* Same array, same bits, but different precision. In general, arbitrary smooth decimals can’t be represented exactly using float32 precision.\n", + "* The index type (Float64Index) actually comes from Pandas (pandas.Float64Index — pandas 1.3.3 documentation). In this case, promoting a float32 type to a float64 in the context of indexing should not affect selection operations, since float32 can safely be cast to float64\n", + "* The number 10.45 is interpreted differently if it is float32 vs float64.\n", + "* Builtin python floats are 64-bit, so IMO Xarray does the right thing by promoting everything to float64 when making comparisons with your lat values.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "import xcdat\n", + "import cdms2\n", + "import cdutil" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": {}, + "outputs": [], + "source": [ + "fn = \"/p/user_pub/climate_work/pochedley1/surface/gistemp1200_GHCNv4_ERSSTv5.nc\"\n", + "\n", + "ds_xcdat = xcdat.open_dataset(fn)\n", + "ds_cdat = cdms2.open(fn)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1. Check dtype for the variable `\"tempanomaly\"`" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dtype('float32')" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "\n", + "ds_xcdat[\"tempanomaly\"].dtype" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dtype('float32')" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "\n", + "ds_cdat(\"tempanomaly\").dtype" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### The `dtype` of the variable is `float32` for both libraries, so they agree." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2. Check dtype for the spatial averaging output for `\"tempanomaly\"`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "xCDAT" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "ds_xcdat_avg = ds_xcdat.spatial.average(\"tempanomaly\", axis=[\"Y\"])\n", + "ta_xcdat_avg = ds_xcdat_avg[\"tempanomaly\"]\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
<xarray.DataArray 'tempanomaly' (time: 1706, lon: 180)>\n", + "array([[ 0.0355457 , 0.02695628, 0.01610215, ..., -0.03079494,\n", + " -0.00960354, 0.00881284],\n", + " [ 0.06634343, 0.04435139, 0.01892282, ..., 0.02397297,\n", + " 0.03883512, 0.05697105],\n", + " [ 0.14746004, 0.11239842, 0.07544763, ..., 0.04206945,\n", + " 0.07549783, 0.11411799],\n", + " ...,\n", + " [ 0.8559086 , 0.81907433, 0.75153327, ..., 1.0020294 ,\n", + " 0.9598275 , 0.90502965],\n", + " [ 0.64544296, 0.6352519 , 0.5955443 , ..., 0.8723912 ,\n", + " 0.82596165, 0.80084634],\n", + " [ 0.8317095 , 0.8324741 , 0.809845 , ..., 0.9373833 ,\n", + " 0.9137498 , 0.8913162 ]], dtype=float32)\n", + "Coordinates:\n", + " * lon (lon) float32 -179.0 -177.0 -175.0 -173.0 ... 175.0 177.0 179.0\n", + " * time (time) datetime64[ns] 1880-01-15 1880-02-15 ... 2022-02-15\n", + "Attributes:\n", + " long_name: Surface temperature anomaly\n", + " units: K\n", + " cell_methods: time: mean
<xarray.Dataset>\n","Dimensions: (lat: 1, lon: 1, time: 5, bnds: 2)\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0\n"," * time (time) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n","Dimensions without coordinates: bnds\n","Data variables:\n"," time_bnds (time, bnds) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n"," ts (time, lat, lon) float64 2.0 nan 1.0 1.0 2.0
<xarray.DataArray 'time_bnds' (time: 5)>\n","array([0.5, 1. , 1. , 1. , 0.5])\n","Coordinates:\n"," * time (time) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n"," month (time) int64 1 2 3 4 1
<xarray.DataArray (month: 4, lat: 1, lon: 1)>\n","array([[[2.]],\n","\n"," [[0.]],\n","\n"," [[1.]],\n","\n"," [[1.]]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0\n"," * month (month) int64 1 2 3 4
<xarray.DataArray 'ts' (lat: 1, lon: 1)>\n","array([[1.33333333]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0
<xarray.DataArray 'time_bnds' (time: 5)>\n","array([0.5, nan, 1. , 1. , 0.5])\n","Coordinates:\n"," * time (time) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n"," month (time) int64 1 2 3 4 1
<xarray.DataArray (month: 4, lat: 1, lon: 1)>\n","array([[[2.]],\n","\n"," [[0.]],\n","\n"," [[1.]],\n","\n"," [[1.]]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0\n"," * month (month) int64 1 2 3 4
<xarray.DataArray 'time_bnds' (time: 5)>\n","array([0.5, 0. , 1. , 1. , 0.5])\n","Coordinates:\n"," * time (time) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n"," month (time) int64 1 2 3 4 1
<xarray.DataArray (month: 4, lat: 1, lon: 1)>\n","array([[[2.]],\n","\n"," [[0.]],\n","\n"," [[1.]],\n","\n"," [[1.]]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0\n"," * month (month) int64 1 2 3 4
<xarray.DataArray 'ts' (lat: 1, lon: 1)>\n","array([[1.33333333]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0
<xarray.DataArray (time: 5, lat: 1, lon: 1)>\n","array([[[ 1.]],\n","\n"," [[nan]],\n","\n"," [[ 1.]],\n","\n"," [[ 1.]],\n","\n"," [[ 1.]]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0\n"," * time (time) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n"," month (time) int64 1 2 3 4 1
<xarray.DataArray (time: 5, lat: 1, lon: 1)>\n","array([[[ 1.]],\n","\n"," [[nan]],\n","\n"," [[ 1.]],\n","\n"," [[ 1.]],\n","\n"," [[ 1.]]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0\n"," * time (time) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n"," month (time) int64 1 2 3 4 1
<xarray.DataArray (time: 5, lat: 1, lon: 1)>\n","array([[[ 1.]],\n","\n"," [[nan]],\n","\n"," [[ 1.]],\n","\n"," [[ 1.]],\n","\n"," [[ 1.]]])\n","Coordinates:\n"," * lat (lat) int64 -90\n"," * lon (lon) int64 0\n"," * time (time) datetime64[ns] 2000-01-01 2000-02-01 ... 2001-01-01\n"," month (time) int64 1 2 3 4 1