validation_and_test_sets.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "hMqWDc_m6rUC"
   },
   "outputs": [],
   "source": [
    "#@title Copyright 2020 Google LLC. Double-click here for license information.\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "# https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4f3CKqFUqL2-"
   },
   "source": [
    "# Validation Sets and Test Sets\n",
    "\n",
    "The previous Colab exercises evaluated the trained model against the training set, which does not provide a strong signal about the quality of your model. In this Colab, you'll experiment with validation sets and test sets.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3spZH_kNkWWX"
   },
   "source": [
    "## Learning objectives\n",
    "\n",
    "After doing this Colab, you'll know how to do the following:\n",
    "\n",
    "  * Split a [training set](https://developers.google.com/machine-learning/glossary/#training_set) into a smaller training set and a [validation set](https://developers.google.com/machine-learning/glossary/#validation_set).\n",
    "  * Analyze deltas between training set and validation set results.\n",
    "  * Test the trained model with a [test set](https://developers.google.com/machine-learning/glossary/#test_set) to determine whether your trained model is [overfitting](https://developers.google.com/machine-learning/glossary/#overfitting).\n",
    "  * Detect and fix a common training problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "gV82DJO3kWpk"
   },
   "source": [
    "## The dataset\n",
    "\n",
    "As in the previous exercise, this exercise uses the [California Housing dataset](https://developers.google.com/machine-learning/crash-course/california-housing-data-description) to predict the `median_house_value` at the city block level.  Like many \"famous\" datasets, the California Housing Dataset actually consists of two separate datasets, each living in separate .csv files:\n",
    "\n",
    "* The training set is in `california_housing_train.csv`.\n",
    "* The test set is in `california_housing_test.csv`.\n",
    "\n",
    "You'll create the validation set by dividing the downloaded training set into two parts:\n",
    "\n",
    "* a smaller training set  \n",
    "* a validation set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "u84mXopntPFZ"
   },
   "source": [
    "## Use the right version of TensorFlow\n",
    "\n",
    "The following hidden code cell ensures that the Colab will run on TensorFlow 2.X."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "FBhNIdUatOU6"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UsageError: Line magic function `%tensorflow_version` not found.\n"
     ]
    }
   ],
   "source": [
    "#@title Run on TensorFlow 2.x\n",
    "%tensorflow_version 2.x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "S8gm6BpqRRuh"
   },
   "source": [
    "## Import relevant modules\n",
    "\n",
    "As before, this first code cell imports the necessary modules and sets a few display options."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "9D8GgUovHbG0"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ERROR:root:Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n",
      "ERROR:root:Internal Python error in the inspect module.\n",
      "Below is the traceback from this internal error.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3331, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"<ipython-input-2-4a80595f3fb0>\", line 4, in <module>\n",
      "    import tensorflow as tf\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 101, in <module>\n",
      "    from tensorflow_core import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\__init__.py\", line 40, in <module>\n",
      "    from tensorflow.python.tools import module_util as _module_util\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\__init__.py\", line 49, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 74, in <module>\n",
      "    raise ImportError(msg)\n",
      "ImportError: Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "\n",
      "Failed to load the native TensorFlow runtime.\n",
      "\n",
      "See https://www.tensorflow.org/install/errors\n",
      "\n",
      "for some common reasons and solutions.  Include the entire stack trace\n",
      "above this error message when asking for help.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2044, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'ImportError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1148, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 316, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 350, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 1490, in getinnerframes\n",
      "    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 1448, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 696, in getsourcefile\n",
      "    if getattr(getmodule(object, filename), '__loader__', None) is not None:\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 733, in getmodule\n",
      "    if ismodule(module) and hasattr(module, '__file__'):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"<frozen importlib._bootstrap>\", line 994, in _gcd_import\n",
      "  File \"<frozen importlib._bootstrap>\", line 971, in _find_and_load\n",
      "  File \"<frozen importlib._bootstrap>\", line 941, in _find_and_load_unlocked\n",
      "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
      "  File \"<frozen importlib._bootstrap>\", line 994, in _gcd_import\n",
      "  File \"<frozen importlib._bootstrap>\", line 971, in _find_and_load\n",
      "  File \"<frozen importlib._bootstrap>\", line 955, in _find_and_load_unlocked\n",
      "  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked\n",
      "  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n",
      "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\__init__.py\", line 42, in <module>\n",
      "    from . _api.v2 import audio\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\_api\\v2\\audio\\__init__.py\", line 10, in <module>\n",
      "    from tensorflow.python.ops.gen_audio_ops import decode_wav\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\ops\\gen_audio_ops.py\", line 9, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\__init__.py\", line 49, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 74, in <module>\n",
      "    raise ImportError(msg)\n",
      "ImportError: Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3331, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"<ipython-input-2-4a80595f3fb0>\", line 4, in <module>\n",
      "    import tensorflow as tf\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 101, in <module>\n",
      "    from tensorflow_core import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\__init__.py\", line 40, in <module>\n",
      "    from tensorflow.python.tools import module_util as _module_util\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\__init__.py\", line 49, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 74, in <module>\n",
      "    raise ImportError(msg)\n",
      "ImportError: Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "\n",
      "Failed to load the native TensorFlow runtime.\n",
      "\n",
      "See https://www.tensorflow.org/install/errors\n",
      "\n",
      "for some common reasons and solutions.  Include the entire stack trace\n",
      "above this error message when asking for help.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2044, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'ImportError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "\n",
      "Failed to load the native TensorFlow runtime.\n",
      "\n",
      "See https://www.tensorflow.org/install/errors\n",
      "\n",
      "for some common reasons and solutions.  Include the entire stack trace\n",
      "above this error message when asking for help.\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3331, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"<ipython-input-2-4a80595f3fb0>\", line 4, in <module>\n",
      "    import tensorflow as tf\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 101, in <module>\n",
      "    from tensorflow_core import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\__init__.py\", line 40, in <module>\n",
      "    from tensorflow.python.tools import module_util as _module_util\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\__init__.py\", line 49, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 74, in <module>\n",
      "    raise ImportError(msg)\n",
      "ImportError: Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "\n",
      "Failed to load the native TensorFlow runtime.\n",
      "\n",
      "See https://www.tensorflow.org/install/errors\n",
      "\n",
      "for some common reasons and solutions.  Include the entire stack trace\n",
      "above this error message when asking for help.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2044, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'ImportError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3254, in run_ast_nodes\n",
      "    if (await self.run_code(code, result,  async_=asy)):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3348, in run_code\n",
      "    self.showtraceback(running_compiled_code=True)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2047, in showtraceback\n",
      "    value, tb, tb_offset=tb_offset)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1415, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1315, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1183, in structured_traceback\n",
      "    formatted_exceptions += self.prepare_chained_exception_message(evalue.__cause__)\n",
      "TypeError: must be str, not list\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2044, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'TypeError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1148, in get_records\n",
      "    return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 316, in wrapped\n",
      "    return f(*args, **kwargs)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 350, in _fixed_getinnerframes\n",
      "    records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 1490, in getinnerframes\n",
      "    frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 1448, in getframeinfo\n",
      "    filename = getsourcefile(frame) or getfile(frame)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 696, in getsourcefile\n",
      "    if getattr(getmodule(object, filename), '__loader__', None) is not None:\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\inspect.py\", line 733, in getmodule\n",
      "    if ismodule(module) and hasattr(module, '__file__'):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"<frozen importlib._bootstrap>\", line 994, in _gcd_import\n",
      "  File \"<frozen importlib._bootstrap>\", line 971, in _find_and_load\n",
      "  File \"<frozen importlib._bootstrap>\", line 941, in _find_and_load_unlocked\n",
      "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
      "  File \"<frozen importlib._bootstrap>\", line 994, in _gcd_import\n",
      "  File \"<frozen importlib._bootstrap>\", line 971, in _find_and_load\n",
      "  File \"<frozen importlib._bootstrap>\", line 955, in _find_and_load_unlocked\n",
      "  File \"<frozen importlib._bootstrap>\", line 665, in _load_unlocked\n",
      "  File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module\n",
      "  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\__init__.py\", line 42, in <module>\n",
      "    from . _api.v2 import audio\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\_api\\v2\\audio\\__init__.py\", line 10, in <module>\n",
      "    from tensorflow.python.ops.gen_audio_ops import decode_wav\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\ops\\gen_audio_ops.py\", line 9, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\__init__.py\", line 49, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 74, in <module>\n",
      "    raise ImportError(msg)\n",
      "ImportError: Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3331, in run_code\n",
      "    exec(code_obj, self.user_global_ns, self.user_ns)\n",
      "  File \"<ipython-input-2-4a80595f3fb0>\", line 4, in <module>\n",
      "    import tensorflow as tf\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 101, in <module>\n",
      "    from tensorflow_core import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\__init__.py\", line 40, in <module>\n",
      "    from tensorflow.python.tools import module_util as _module_util\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 50, in __getattr__\n",
      "    module = self._load()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow\\__init__.py\", line 44, in _load\n",
      "    module = _importlib.import_module(self.__name__)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\importlib\\__init__.py\", line 126, in import_module\n",
      "    return _bootstrap._gcd_import(name[level:], package, level)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\__init__.py\", line 49, in <module>\n",
      "    from tensorflow.python import pywrap_tensorflow\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 74, in <module>\n",
      "    raise ImportError(msg)\n",
      "ImportError: Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "\n",
      "Failed to load the native TensorFlow runtime.\n",
      "\n",
      "See https://www.tensorflow.org/install/errors\n",
      "\n",
      "for some common reasons and solutions.  Include the entire stack trace\n",
      "above this error message when asking for help.\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2044, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'ImportError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3254, in run_ast_nodes\n",
      "    if (await self.run_code(code, result,  async_=asy)):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 3348, in run_code\n",
      "    self.showtraceback(running_compiled_code=True)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2047, in showtraceback\n",
      "    value, tb, tb_offset=tb_offset)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1415, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1315, in structured_traceback\n",
      "    self, etype, value, tb, tb_offset, number_of_lines_of_context\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\", line 1183, in structured_traceback\n",
      "    formatted_exceptions += self.prepare_chained_exception_message(evalue.__cause__)\n",
      "TypeError: must be str, not list\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\", line 2044, in showtraceback\n",
      "    stb = value._render_traceback_()\n",
      "AttributeError: 'TypeError' object has no attribute '_render_traceback_'\n",
      "\n",
      "During handling of the above exception, another exception occurred:\n",
      "\n",
      "Traceback (most recent call last):\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\", line 58, in <module>\n",
      "    from tensorflow.python.pywrap_tensorflow_internal import *\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 28, in <module>\n",
      "    _pywrap_tensorflow_internal = swig_import_helper()\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\", line 24, in swig_import_helper\n",
      "    _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 243, in load_module\n",
      "    return load_dynamic(name, filename, file)\n",
      "  File \"C:\\Users\\David\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\", line 343, in load_dynamic\n",
      "    return _load(spec)\n",
      "ImportError: DLL load failed: The specified module could not be found.\n",
      "\n",
      "\n",
      "Failed to load the native TensorFlow runtime.\n",
      "\n",
      "See https://www.tensorflow.org/install/errors\n",
      "\n",
      "for some common reasons and solutions.  Include the entire stack trace\n",
      "above this error message when asking for help.\n"
     ]
    },
    {
     "ename": "TypeError",
     "evalue": "must be str, not list",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     57\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 58\u001b[1;33m   \u001b[1;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpywrap_tensorflow_internal\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[1;33m*\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     59\u001b[0m   \u001b[1;32mfrom\u001b[0m \u001b[0mtensorflow\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpython\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mpywrap_tensorflow_internal\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0m__version__\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     27\u001b[0m             \u001b[1;32mreturn\u001b[0m \u001b[0m_mod\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 28\u001b[1;33m     \u001b[0m_pywrap_tensorflow_internal\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mswig_import_helper\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     29\u001b[0m     \u001b[1;32mdel\u001b[0m \u001b[0mswig_import_helper\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\tensorflow_core\\python\\pywrap_tensorflow_internal.py\u001b[0m in \u001b[0;36mswig_import_helper\u001b[1;34m()\u001b[0m\n\u001b[0;32m     23\u001b[0m             \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 24\u001b[1;33m                 \u001b[0m_mod\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mimp\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mload_module\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'_pywrap_tensorflow_internal'\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfp\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpathname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mdescription\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     25\u001b[0m             \u001b[1;32mfinally\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\u001b[0m in \u001b[0;36mload_module\u001b[1;34m(name, file, filename, details)\u001b[0m\n\u001b[0;32m    242\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 243\u001b[1;33m             \u001b[1;32mreturn\u001b[0m \u001b[0mload_dynamic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mname\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfilename\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mfile\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    244\u001b[0m     \u001b[1;32melif\u001b[0m \u001b[0mtype_\u001b[0m \u001b[1;33m==\u001b[0m \u001b[0mPKG_DIRECTORY\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\imp.py\u001b[0m in \u001b[0;36mload_dynamic\u001b[1;34m(name, path, file)\u001b[0m\n\u001b[0;32m    342\u001b[0m             name=name, loader=loader, origin=path)\n\u001b[1;32m--> 343\u001b[1;33m         \u001b[1;32mreturn\u001b[0m \u001b[0m_load\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mspec\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    344\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mImportError\u001b[0m: DLL load failed: The specified module could not be found.",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\u001b[0m in \u001b[0;36mshowtraceback\u001b[1;34m(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)\u001b[0m\n\u001b[0;32m   2043\u001b[0m                         \u001b[1;31m# in the engines. This should return a list of strings.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 2044\u001b[1;33m                         \u001b[0mstb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_render_traceback_\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   2045\u001b[0m                     \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mAttributeError\u001b[0m: 'ImportError' object has no attribute '_render_traceback_'",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\u001b[0m in \u001b[0;36mrun_code\u001b[1;34m(self, code_obj, result, async_)\u001b[0m\n\u001b[0;32m   3346\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3347\u001b[0m                 \u001b[0mresult\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0merror_in_exec\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msys\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mexc_info\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;36m1\u001b[0m\u001b[1;33m]\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 3348\u001b[1;33m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mshowtraceback\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mrunning_compiled_code\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;32mTrue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   3349\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   3350\u001b[0m             \u001b[0moutflag\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mFalse\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\interactiveshell.py\u001b[0m in \u001b[0;36mshowtraceback\u001b[1;34m(self, exc_tuple, filename, tb_offset, exception_only, running_compiled_code)\u001b[0m\n\u001b[0;32m   2045\u001b[0m                     \u001b[1;32mexcept\u001b[0m \u001b[0mException\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2046\u001b[0m                         stb = self.InteractiveTB.structured_traceback(etype,\n\u001b[1;32m-> 2047\u001b[1;33m                                             value, tb, tb_offset=tb_offset)\n\u001b[0m\u001b[0;32m   2048\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   2049\u001b[0m                     \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_showtraceback\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0metype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mstb\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\u001b[0m in \u001b[0;36mstructured_traceback\u001b[1;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[0;32m   1413\u001b[0m             \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mtb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mtb\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1414\u001b[0m         return FormattedTB.structured_traceback(\n\u001b[1;32m-> 1415\u001b[1;33m             self, etype, value, tb, tb_offset, number_of_lines_of_context)\n\u001b[0m\u001b[0;32m   1416\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1417\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\u001b[0m in \u001b[0;36mstructured_traceback\u001b[1;34m(self, etype, value, tb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[0;32m   1313\u001b[0m             \u001b[1;31m# Verbose modes need a full traceback\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1314\u001b[0m             return VerboseTB.structured_traceback(\n\u001b[1;32m-> 1315\u001b[1;33m                 \u001b[0mself\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0metype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mvalue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtb\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mtb_offset\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mnumber_of_lines_of_context\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1316\u001b[0m             )\n\u001b[0;32m   1317\u001b[0m         \u001b[1;32melif\u001b[0m \u001b[0mmode\u001b[0m \u001b[1;33m==\u001b[0m \u001b[1;34m'Minimal'\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32m~\\code\\python\\anaconda3\\envs\\mlcc\\lib\\site-packages\\IPython\\core\\ultratb.py\u001b[0m in \u001b[0;36mstructured_traceback\u001b[1;34m(self, etype, evalue, etb, tb_offset, number_of_lines_of_context)\u001b[0m\n\u001b[0;32m   1181\u001b[0m         \u001b[0mexception\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mget_parts_of_chained_exception\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mevalue\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1182\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mexception\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m-> 1183\u001b[1;33m             \u001b[0mformatted_exceptions\u001b[0m \u001b[1;33m+=\u001b[0m \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mprepare_chained_exception_message\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mevalue\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m__cause__\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m   1184\u001b[0m             \u001b[0metype\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mevalue\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0metb\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mexception\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m   1185\u001b[0m         \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: must be str, not list"
     ]
    }
   ],
   "source": [
    "#@title Import modules\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "pd.options.display.max_rows = 10\n",
    "pd.options.display.float_format = \"{:.1f}\".format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xjvrrClQeAJu"
   },
   "source": [
    "## Load the datasets from the internet\n",
    "\n",
    "The following code cell loads the separate .csv files and creates the following two pandas DataFrames:\n",
    "\n",
    "* `train_df`, which contains the training set.\n",
    "* `test_df`, which contains the test set.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zUnTc_wfd_o3"
   },
   "outputs": [],
   "source": [
    "train_df = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_train.csv\")\n",
    "test_df = pd.read_csv(\"https://download.mlcc.google.com/mledu-datasets/california_housing_test.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P_KBdj2M_yjM"
   },
   "source": [
    "## Scale the label values\n",
    "\n",
    "The following code cell scales the `median_house_value`. \n",
    "See the previous Colab exercise for details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3hc7QQhaAFXD"
   },
   "outputs": [],
   "source": [
    "scale_factor = 1000.0\n",
    "\n",
    "# Scale the training set's label.\n",
    "train_df[\"median_house_value\"] /= scale_factor \n",
    "\n",
    "# Scale the test set's label\n",
    "test_df[\"median_house_value\"] /= scale_factor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FhessIIV8VPc"
   },
   "source": [
    "## Load the functions that build and train a model\n",
    "\n",
    "The following code cell defines two functions:\n",
    "\n",
    "  * `build_model`, which defines the model's topography.\n",
    "  * `train_model`, which will ultimately train the model, outputting not only the loss value for the training set but also the loss value for the validation set. \n",
    "\n",
    "Since you don't need to understand model building code right now, we've hidden this code cell. As always, you must run hidden code cells."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "bvonhK857msj"
   },
   "outputs": [],
   "source": [
    "#@title Define the functions that build and train a model\n",
    "def build_model(my_learning_rate):\n",
    "  \"\"\"Create and compile a simple linear regression model.\"\"\"\n",
    "  # Most simple tf.keras models are sequential.\n",
    "  model = tf.keras.models.Sequential()\n",
    "\n",
    "  # Add one linear layer to the model to yield a simple linear regressor.\n",
    "  model.add(tf.keras.layers.Dense(units=1, input_shape=(1,)))\n",
    "\n",
    "  # Compile the model topography into code that TensorFlow can efficiently\n",
    "  # execute. Configure training to minimize the model's mean squared error. \n",
    "  model.compile(optimizer=tf.keras.optimizers.RMSprop(lr=my_learning_rate),\n",
    "                loss=\"mean_squared_error\",\n",
    "                metrics=[tf.keras.metrics.RootMeanSquaredError()])\n",
    "\n",
    "  return model               \n",
    "\n",
    "\n",
    "def train_model(model, df, feature, label, my_epochs, \n",
    "                my_batch_size=None, my_validation_split=0.1):\n",
    "  \"\"\"Feed a dataset into the model in order to train it.\"\"\"\n",
    "\n",
    "  history = model.fit(x=df[feature],\n",
    "                      y=df[label],\n",
    "                      batch_size=my_batch_size,\n",
    "                      epochs=my_epochs,\n",
    "                      validation_split=my_validation_split)\n",
    "\n",
    "  # Gather the model's trained weight and bias.\n",
    "  trained_weight = model.get_weights()[0]\n",
    "  trained_bias = model.get_weights()[1]\n",
    "\n",
    "  # The list of epochs is stored separately from the \n",
    "  # rest of history.\n",
    "  epochs = history.epoch\n",
    "  \n",
    "  # Isolate the root mean squared error for each epoch.\n",
    "  hist = pd.DataFrame(history.history)\n",
    "  rmse = hist[\"root_mean_squared_error\"]\n",
    "\n",
    "  return epochs, rmse, history.history   \n",
    "\n",
    "print(\"Defined the build_model and train_model functions.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8gRu4Ri0D8tH"
   },
   "source": [
    "## Define plotting functions\n",
    "\n",
    "The `plot_the_loss_curve` function plots loss vs. epochs for both the training set and the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "QA7hsqPZDvVM"
   },
   "outputs": [],
   "source": [
    "#@title Define the plotting function\n",
    "\n",
    "def plot_the_loss_curve(epochs, mae_training, mae_validation):\n",
    "  \"\"\"Plot a curve of loss vs. epoch.\"\"\"\n",
    "\n",
    "  plt.figure()\n",
    "  plt.xlabel(\"Epoch\")\n",
    "  plt.ylabel(\"Root Mean Squared Error\")\n",
    "\n",
    "  plt.plot(epochs[1:], mae_training[1:], label=\"Training Loss\")\n",
    "  plt.plot(epochs[1:], mae_validation[1:], label=\"Validation Loss\")\n",
    "  plt.legend()\n",
    "  \n",
    "  # We're not going to plot the first epoch, since the loss on the first epoch\n",
    "  # is often substantially greater than the loss for other epochs.\n",
    "  merged_mae_lists = mae_training[1:] + mae_validation[1:]\n",
    "  highest_loss = max(merged_mae_lists)\n",
    "  lowest_loss = min(merged_mae_lists)\n",
    "  delta = highest_loss - lowest_loss\n",
    "  print(delta)\n",
    "\n",
    "  top_of_y_axis = highest_loss + (delta * 0.05)\n",
    "  bottom_of_y_axis = lowest_loss - (delta * 0.05)\n",
    "   \n",
    "  plt.ylim([bottom_of_y_axis, top_of_y_axis])\n",
    "  plt.show()  \n",
    "\n",
    "print(\"Defined the plot_the_loss_curve function.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jipBqEQXlsN8"
   },
   "source": [
    "## Task 1: Experiment with the validation split\n",
    "\n",
    "In the following code cell, you'll see a variable named `validation_split`, which we've initialized at 0.2.  The `validation_split` variable specifies the proportion of the original training set that will serve as the validation set. The original training set contains 17,000 examples. Therefore, a `validation_split` of 0.2 means that:\n",
    "\n",
    "* 17,000 * 0.2 ~= 3,400 examples will become the validation set.\n",
    "* 17,000 * 0.8 ~= 13,600 examples will become the new training set.\n",
    "\n",
    "The following code builds a model, trains it on the training set, and evaluates the built model on both:\n",
    "\n",
    "* The training set.\n",
    "* And the validation set.\n",
    "\n",
    "If the data in the training set is similar to the data in the validation set, then the two loss curves and the final loss values should be almost identical. However, the loss curves and final loss values are **not** almost identical. Hmm, that's odd.  \n",
    "\n",
    "Experiment with two or three different values of `validation_split`.  Do different values of `validation_split` fix the problem? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "knP23Taoa00a"
   },
   "outputs": [],
   "source": [
    "# The following variables are the hyperparameters.\n",
    "learning_rate = 0.08\n",
    "epochs = 30\n",
    "batch_size = 100\n",
    "\n",
    "# Split the original training set into a reduced training set and a\n",
    "# validation set. \n",
    "validation_split=0.2\n",
    "\n",
    "# Identify the feature and the label.\n",
    "my_feature=\"median_income\"  # the median income on a specific city block.\n",
    "my_label=\"median_house_value\" # the median value of a house on a specific city block.\n",
    "# That is, you're going to create a model that predicts house value based \n",
    "# solely on the neighborhood's median income.  \n",
    "\n",
    "# Discard any pre-existing version of the model.\n",
    "my_model = None\n",
    "\n",
    "# Invoke the functions to build and train the model.\n",
    "my_model = build_model(learning_rate)\n",
    "epochs, rmse, history = train_model(my_model, train_df, my_feature, \n",
    "                                    my_label, epochs, batch_size, \n",
    "                                    validation_split)\n",
    "\n",
    "plot_the_loss_curve(epochs, history[\"root_mean_squared_error\"], \n",
    "                    history[\"val_root_mean_squared_error\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TKa11JK4Pm3f"
   },
   "source": [
    "## Task 2: Determine **why** the loss curves differ\n",
    "\n",
    "No matter how you split the training set and the validation set, the loss curves differ significantly. Evidently, the data in the training set isn't similar enough to the data in the validation set. Counterintuitive? Yes, but this problem is actually pretty common in machine learning. \n",
    "\n",
    "Your task is to determine **why** the loss curves aren't highly similar. As with most issues in machine learning, the problem is rooted in the data itself. To solve this mystery of why the training set and validation set aren't almost identical, write a line or two of [pandas code](https://colab.research.google.com/drive/1gUeYFsYmoyqpQJWq7krrZZNUFvBPwrJf) in the following code cell.  Here are a couple of hints:\n",
    "\n",
    "  * The previous code cell split the original training set into:\n",
    "    * a reduced training set (the original training set - the validation set)\n",
    "    * the validation set \n",
    "  * By default, the pandas [`head`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.head.html) method outputs the *first* 5 rows of the DataFrame. To see more of the training set, specify the `n` argument to `head` and assign a large positive integer to `n`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VJQcAZkwJt_p"
   },
   "outputs": [],
   "source": [
    "# Write some code in this code cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "EnNvkFwwK8WY"
   },
   "outputs": [],
   "source": [
    "#@title Double-click for a possible solution to Task 2.\n",
    "\n",
    "# Examine examples 0 through 4 and examples 25 through 29\n",
    "# of the training set\n",
    "train_df.head(n=1000)\n",
    "\n",
    "# The original training set is sorted by longitude. \n",
    "# Apparently, longitude influences the relationship of\n",
    "# total_rooms to median_house_value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "rw4xI1ZEckI8"
   },
   "source": [
    "## Task 3. Fix the problem\n",
    "\n",
    "To fix the problem, shuffle the examples in the training set before splitting the examples into a training set and validation set. To do so, take the following steps:\n",
    "\n",
    "1. Shuffle the data in the training set by adding the following line anywhere before you call `train_model` (in the code cell associated with Task 1):\n",
    "\n",
    "```\n",
    "  shuffled_train_df = train_df.reindex(np.random.permutation(train_df.index))\n",
    "```                                    \n",
    "\n",
    "2. Pass `shuffled_train_df` (instead of `train_df`) as the second argument to `train_model` (in the code call associated with Task 1) so that the call becomes as follows:\n",
    "\n",
    "```\n",
    "  epochs, rmse, history = train_model(my_model, shuffled_train_df, my_feature, \n",
    "                                      my_label, epochs, batch_size, \n",
    "                                      validation_split)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "ncODhpv0h-LG"
   },
   "outputs": [],
   "source": [
    "#@title Double-click to view the complete implementation.\n",
    "\n",
    "# The following variables are the hyperparameters.\n",
    "learning_rate = 0.08\n",
    "epochs = 70\n",
    "batch_size = 100\n",
    "\n",
    "# Split the original training set into a reduced training set and a\n",
    "# validation set. \n",
    "validation_split=0.2\n",
    "\n",
    "# Identify the feature and the label.\n",
    "my_feature=\"median_income\"  # the median income on a specific city block.\n",
    "my_label=\"median_house_value\" # the median value of a house on a specific city block.\n",
    "# That is, you're going to create a model that predicts house value based \n",
    "# solely on the neighborhood's median income.  \n",
    "\n",
    "# Discard any pre-existing version of the model.\n",
    "my_model = None\n",
    "\n",
    "# Shuffle the examples.\n",
    "shuffled_train_df = train_df.reindex(np.random.permutation(train_df.index)) \n",
    "\n",
    "# Invoke the functions to build and train the model. Train on the shuffled\n",
    "# training set.\n",
    "my_model = build_model(learning_rate)\n",
    "epochs, rmse, history = train_model(my_model, shuffled_train_df, my_feature, \n",
    "                                    my_label, epochs, batch_size, \n",
    "                                    validation_split)\n",
    "\n",
    "plot_the_loss_curve(epochs, history[\"root_mean_squared_error\"], \n",
    "                    history[\"val_root_mean_squared_error\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tKN239_miW8C"
   },
   "source": [
    "Experiment with `validation_split` to answer the following questions:\n",
    "\n",
    "* With the training set shuffled, is the final loss for the training set closer to the final loss for the validation set?  \n",
    "* At what range of values of `validation_split` do the final loss values for the training set and validation set diverge meaningfully?  Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "-UAJ3Q86iz31"
   },
   "outputs": [],
   "source": [
    "#@title Double-click for the answers to the questions\n",
    "\n",
    "# Yes, after shuffling the original training set, \n",
    "# the final loss for the training set and the \n",
    "# validation set become much closer.\n",
    "\n",
    "# If validation_split < 0.15,\n",
    "# the final loss values for the training set and\n",
    "# validation set diverge meaningfully.  Apparently,\n",
    "# the validation set no longer contains enough examples. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1PP-O8TOZOeo"
   },
   "source": [
    "## Task 4: Use the Test Dataset to Evaluate Your Model's Performance\n",
    "\n",
    "The test set usually acts as the ultimate judge of a model's quality. The test set can serve as an impartial judge because its examples haven't been used in training the model. Run the following code cell to evaluate the model with the test set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nd_Sw2cygOip"
   },
   "outputs": [],
   "source": [
    "x_test = test_df[my_feature]\n",
    "y_test = test_df[my_label]\n",
    "\n",
    "results = my_model.evaluate(x_test, y_test, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qoyQKvsjmV_A"
   },
   "source": [
    "Compare the root mean squared error of the model when evaluated on each of the three datasets:\n",
    "\n",
    "* training set: look for `root_mean_squared_error` in the final training epoch.\n",
    "* validation set: look for `val_root_mean_squared_error` in the final training epoch.\n",
    "* test set: run the preceding code cell and examine the `root_mean_squred_error`.\n",
    "\n",
    "Ideally, the root mean squared error of all three sets should be similar. Are they?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "cellView": "form",
    "colab": {},
    "colab_type": "code",
    "id": "FxXtp-aVdIgJ"
   },
   "outputs": [],
   "source": [
    "#@title Double-click for an answer\n",
    "\n",
    "# In our experiments, yes, the rmse values \n",
    "# were similar enough. "
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Validation and Test Sets.ipynb",
   "private_outputs": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}