06-01.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Lecture 9 - Files\n",
    "\n",
    "## Week 6 Monday\n",
    "\n",
    "## Miles Chen, PhD\n",
    "\n",
    "Adapted from Chapter 14 of Think Python by Allen B Downey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Persistence\n",
    "\n",
    "Most of the code we have run so far are transient. They will run during your session and after producing some output, their data disappears.\n",
    "\n",
    "Other programs need to be **persistent**, meaning that they need to store (at least some of) their data.\n",
    "\n",
    "This chapter will focus on reading and writing files to safe data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Opening a file\n",
    "\n",
    "You can open a file with `open()`\n",
    "\n",
    "You can write a to a file using `'w'` as a second parameter: `open('filename.txt', 'w')`\n",
    "\n",
    "If the file exists, running this command will replace the file. If the file does not exist, it will be created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fout = open('output.txt', 'w')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The function `open()` returns a file object that allows you to write to the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line1 = \"This is the first line of text.\\n\"\n",
    "fout.write(line1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The method returns the number of characters that were written."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you call write again, it adds the new data to the end of the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "38"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "line2 = \"This will be the second line of text.\\n\"\n",
    "fout.write(line2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When you are done writing, you should close the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "fout.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Closing the file frees up the system resources dedicated to tracking that file. If you forget to close the file and Python sees no object names bound to the file object, Python's garbage collector will automatically close the connection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## with\n",
    "\n",
    "You can use the `with` keyword to have operations run with a particular file. When the operations finish, Python will automatically close the file. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# opening a file and writing lines to it:\n",
    "with open('output.txt', 'w') as file:\n",
    "    file.write(line1)\n",
    "    file.write(line2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This is the first line of text.\n",
      "\n",
      "This will be the second line of text.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# opening the file and printing each line\n",
    "with open('output.txt', 'r') as file:\n",
    "    for i in file:\n",
    "        print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## The Format Operator\n",
    "\n",
    "When using `write`, the argument has to be a string.\n",
    "\n",
    "The easiest way to convert non-strings to strings is with `str()`. If you don't convert the value to a string, you can run into problems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "unsupported operand type(s) for +: 'float' and 'str'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-7-bd11443d914f>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;36m42.3\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0my\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mx\u001b[0m \u001b[1;33m+\u001b[0m \u001b[1;34m\" is my favorite number\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m: unsupported operand type(s) for +: 'float' and 'str'"
     ]
    }
   ],
   "source": [
    "x = 42.3\n",
    "y = x + \" is my favorite number\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42.3 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "x = 42.3\n",
    "y = str(x) + \" is my favorite number\"\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another option is to use the format operator `%`. This is the same symbol for modulo division, but can be applied to format strings.\n",
    "\n",
    "Python has **format sequences** which can be used to specify the format of the string.\n",
    "\n",
    "See: https://docs.python.org/3/library/stdtypes.html#printf-style-string-formatting\n",
    "\n",
    "For example, `'%d'` will tell Python to format the value as an integer.\n",
    "\n",
    "`'%g'` will tell Python to format the value as a floating point number.\n",
    "\n",
    "The format operator can be placed inline into any string and Python will replace the format sequence with the appropriate string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "x = 42.3\n",
    "y = '%d is my favorite number' % x\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "42.3 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "x = 42.3\n",
    "y = '%g is my favorite number' % x\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The format operator can also be used with multiple placeholders and a tuple of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My age is 85, and 42.3 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "x = 42.3\n",
    "age = 85\n",
    "y = 'My age is %d, and %g is my favorite number' % (age, x)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My age is 85, and 42.3 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "# alternative method using + operator\n",
    "x = 42.3\n",
    "age = 85\n",
    "y = 'My age is ' + str(age) + ', and ' + str(x) + ' is my favorite number'\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My name is Joe Bruin, I am 100 years old, and 83 is my favorite number\n"
     ]
    }
   ],
   "source": [
    "y = 'My name is %s, I am %d years old, and %g is my favorite number' % (\"Joe Bruin\", 100, 83)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Filenames and Paths\n",
    "\n",
    "Python's `os` module allows you to work with your computer's file system in case you need to work with directories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'D:\\\\OneDrive\\\\Teaching\\\\21\\\\2020-sp-stats-21'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Use os.getcwd() to get the current working directory\n",
    "cwd = os.getcwd()\n",
    "cwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "When you provide a filename to Python, it is generally a relative path because it relates to the current working directory.\n",
    "\n",
    "You can ask for absolute paths with `os.path.abspath()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'D:\\\\OneDrive\\\\Teaching\\\\21\\\\2020-sp-stats-21\\\\output.txt'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.abspath('output.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### more os functions\n",
    "\n",
    "You can read more about Python's OS functions at: https://docs.python.org/3/library/os.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if a file exists\n",
    "os.path.exists('memo.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.exists('output.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check if something is a directory\n",
    "os.path.isdir('output.txt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.path.isdir('.git')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['.git',\n",
       " '.gitignore',\n",
       " '.ipynb_checkpoints',\n",
       " '02-01 slides.pdf',\n",
       " '02-01.ipynb',\n",
       " '02-01.slides.html',\n",
       " '02-02 slides.pdf',\n",
       " '02-02.ipynb',\n",
       " '02-02.slides.html',\n",
       " '02-03 slides.pdf',\n",
       " '02-03.ipynb',\n",
       " '02-03.slides.html',\n",
       " '03-01.ipynb',\n",
       " '03-02 slides.pdf',\n",
       " '03-02.ipynb',\n",
       " '03-02.slides.html',\n",
       " '03-03 slides.pdf',\n",
       " '03-03.ipynb',\n",
       " '03-03.slides.html',\n",
       " '04-01 slides.pdf',\n",
       " '04-01.ipynb',\n",
       " '04-01.slides.html',\n",
       " '04-02 slides.pdf',\n",
       " '04-02.ipynb',\n",
       " '04-02.slides.html',\n",
       " '04-03 slides.pdf',\n",
       " '04-03.ipynb',\n",
       " '04-03.slides.html',\n",
       " '05-01 slides.pdf',\n",
       " '05-01.ipynb',\n",
       " '05-01.slides.html',\n",
       " '05-02 slides.pdf',\n",
       " '05-02.ipynb',\n",
       " '05-02.slides.html',\n",
       " '06-01.ipynb',\n",
       " '06-01.slides.html',\n",
       " 'captions.bak',\n",
       " 'captions.dat',\n",
       " 'captions.dir',\n",
       " 'commands.txt',\n",
       " 'LICENSE',\n",
       " 'new.txt',\n",
       " 'objects.pkl',\n",
       " 'output.txt',\n",
       " 'script01.py',\n",
       " 'script02.py']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# list the contents of the current working directory\n",
    "os.listdir(cwd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Catching Exceptions\n",
    "\n",
    "When dealing with files, and also just in programming in general, a lot of things can go wrong and Python will throw an exception.\n",
    "\n",
    "You can often catch many of the exceptions with `try ... except` which works in a manner similar to `if ... else`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'bad_file'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-22-a7d7d7ad396b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mfin\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mopen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'bad_file'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'bad_file'"
     ]
    }
   ],
   "source": [
    "fin = open('bad_file')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Something went wrong.\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    fin = open('bad_file')\n",
    "except:\n",
    "    print('Something went wrong.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Python first tries the commands in the `try:` block. If everything goes well, it skips the commands in the `except:` block. If it encounters an exception in the `try` block, it immediately exits the block and executes the code in the `except` block.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Python dbm databases\n",
    "\n",
    "A Python dbm database is like a Python dictionary but saved to a file on the harddrive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# You first have to import the dbm library\n",
    "import dbm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "`dbm.open()` will open the database connection\n",
    "\n",
    "The 'c' option will create it if it doesn't yet exist. It will not overwrite an existing file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "\n",
    "db = dbm.open('captions', 'c')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Add keys and values much like a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db['cleese.png'] = 'Photo of John Cleese'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The key here is the string 'cleese.png'. It is not a file, just the string of a filename (that may or may not exist). The value is another string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'Photo of John Cleese'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# retrieve values like a dictionary\n",
    "db['cleese.png']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# I add another key-value pair.\n",
    "db['jones.png'] = 'Photo of Terry Jones'\n",
    "# if you need to, you can also delete an entry using del(db['jones.png'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'cleese.png' b'Photo of John Cleese'\n",
      "b'jones.png' b'Photo of Terry Jones'\n"
     ]
    }
   ],
   "source": [
    "# you can iterate through a database\n",
    "for key in db:\n",
    "    print(key, db[key])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# you should close when done\n",
    "db.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "captions.bak\n",
      "captions.dat\n",
      "captions.dir\n"
     ]
    }
   ],
   "source": [
    "# creating the captions dbm will add a few files to your working directory:\n",
    "for i in os.listdir(cwd):\n",
    "    if os.path.isfile(i) and 'captions' in i:\n",
    "        print(i)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Python Pickles\n",
    "\n",
    "A major limitation with dbm is that it can only store strings or byte objects.\n",
    "\n",
    "If you want to store other objects you created in Python, you'll want to `pickle` them. This converts the Python object into a string that can then be written to the dbm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "t = [1 ,2, 3]\n",
    "type(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db = dbm.open('captions', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "values must be bytes or strings",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-34-f05b3aca1a75>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[1;31m# if I try to save the list t to the dbm, it causes an error.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 2\u001b[1;33m \u001b[0mdb\u001b[0m\u001b[1;33m[\u001b[0m\u001b[1;34m'list'\u001b[0m\u001b[1;33m]\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mt\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\dbm\\dumb.py\u001b[0m in \u001b[0;36m__setitem__\u001b[1;34m(self, key, val)\u001b[0m\n\u001b[0;32m    197\u001b[0m             \u001b[0mval\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mval\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mencode\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'utf-8'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    198\u001b[0m         \u001b[1;32melif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mval\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mbytes\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mbytearray\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 199\u001b[1;33m             \u001b[1;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"values must be bytes or strings\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    200\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_verify_open\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    201\u001b[0m         \u001b[0mself\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0m_modified\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mTypeError\u001b[0m: values must be bytes or strings"
     ]
    }
   ],
   "source": [
    "# if I try to save the list t to the dbm, it causes an error.\n",
    "db['list'] = t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# pickle.dumps converts the python object to a string\n",
    "s = pickle.dumps(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x03]q\\x00(K\\x01K\\x02K\\x03e.'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# the string representation of the list t\n",
    "# This is not meant to be human-readable. It is simply a format that can be stored in the dbm.\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db['list'] = s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "db.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now that we have saved the \"pickled\" list to the dbm, let's see if we can retreive it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# open the dbm\n",
    "db = dbm.open('captions', 'c')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "b'\\x80\\x03]q\\x00(K\\x01K\\x02K\\x03e.'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# retrieve the string associated with the key 'list'\n",
    "db['list']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3]"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to conver the string back to a python object, you can use pickle.loads()\n",
    "pickle.loads(db['list'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can pickle objects directly to a pickle file.\n",
    "\n",
    "You can pickle multiple objects by grouping them together in a list or tuple."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "obj0 = ['file1', 'something else']\n",
    "obj1 = [1, 2, 3]\n",
    "obj2 = \"hello world!\"\n",
    "\n",
    "with open('objects.pkl', 'wb') as file:\n",
    "    pickle.dump([obj0, obj1, obj2], file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Loading the pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "with open('objects.pkl', \"rb\") as file:\n",
    "    x1, x2, x3 = pickle.load(file)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['file1', 'something else']"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1, 2, 3]"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'hello world!'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Read more about pickling:\n",
    "\n",
    "https://docs.python.org/3/library/pickle.html#what-can-be-pickled-and-unpickled"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}