resources/tutorials/FetchCHIPapers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bibliography Fetch\n",
    "\n",
    "A couple of the notebooks use the 'CHI Papers' data set.  This notebook creates that data set from the original sources."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "\n",
    "Let's import libraries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import html5lib as html\n",
    "import requests\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fetching the Index\n",
    "\n",
    "The full list of files in the HCI Bibliography is on the [index page](http://hcibib.org/listdir.cgi).  We'll use that to get our file list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "hcibib_root = 'http://hcibib.org'\n",
    "file_index = requests.get(f'{hcibib_root}/listdir.cgi')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx_html = html.parse(file_index.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse the files out of the HTML content itself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1988"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files = {}\n",
    "bib_re = re.compile(r'^/bibdata/(.*\\.bib)')\n",
    "for link in idx_html.findall('*//{http://www.w3.org/1999/xhtml}a'):\n",
    "    href = link.get('href')\n",
    "    m = bib_re.match(href)\n",
    "    if m:\n",
    "        files[m.group(1)] = href\n",
    "len(files)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decoding Data\n",
    "\n",
    "Let's get an example file to see what we're dealing with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "%M C.CHI.10.1.1\n",
      "%T Estimating residual error rate in recognized handwritten documents using\n",
      "artificial error injection\n",
      "%S EPIC #FAIL\n",
      "%A Lank, Edward\n",
      "%A Stedman, Ryan\n",
      "%A Terry, Michael\n",
      "%B Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems\n",
      "%D 2010-04-10\n",
      "%V 1\n",
      "%P 1-4\n",
      "%K artificial error, handwriting recognition, residual error\n",
      "%* (c) Copyright 2010 ACM\n",
      "%W http://doi.acm.org/10.1145/1753326.1753328\n",
      "%X Both handwriting recognition systems and their users are error prone.\n",
      "Handwriting recognizers make recognition errors, and users may miss those\n",
      "errors when verifying output. As a result, it is common for recognized\n",
      "documents to contain residual errors. Unfortunately, in some application\n",
      "domains (e.g. health informatics), tolerance for residual errors in recognized\n",
      "handwriting may be very low, and a desire might exist to maximize user accuracy\n",
      "during verification. In this paper, we present a technique that allows us to\n",
      "measure the performance of a user verifying recognizer output. We inject\n",
      "artificial errors into a set of recognized handwritten forms and show that the\n",
      "rate of injected errors and recognition errors caught is highly correlated in\n",
      "real time. Systems supporting user verification can make use of this measure of\n",
      "user accuracy in a variety of ways. For example, they can force users to slow\n",
      "down or can highlight injected errors that were missed, thus encouraging users\n",
      "to take more care.\n",
      "\n",
      "%M C.CHI.10.1.5\n",
      "%T Predicting the cost of error correction in character-based text entry\n",
      "technologies\n",
      "%S EPIC #FAIL\n",
      "%A Arif, Ahmed Sabbir\n",
      "%A Stuerzlinger, Wolfgang\n",
      "%B Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems\n",
      "%D 2010-04-10\n",
      "%V 1\n",
      "%P 5-14\n",
      "%K cognitive model, error correction, error rate, hand-held devices, mobile\n",
      "phone, performance metric, prediction, text entry\n",
      "%* (c) Copyright 2010 ACM\n",
      "%W http://doi.acm.org/10.1145/1753326.1753329\n",
      "%X Researchers have developed many models to predict and understand human\n",
      "performance in text entry. Most of the models are specific to a technology or\n",
      "fail to account for human factors and variations in system parameters, and the\n",
      "relationship between them. Moreover, the process of fixing errors and its\n",
      "effects on text entry performance has not been studied. Here, we first analyze\n",
      "real-life text entry error correction behaviors. We then use our findings to\n",
      "develop a new model to predict the cost of error correction for character-based\n",
      "text entry technologies. We validate our model against quantities derived from\n",
      "the literature, as well as with a user study. Our study shows that the\n",
      "predicted and observed cost of error correction correspond well. At the end, we\n",
      "discuss potential applications of our new model.\n",
      "\n",
      "%M C.CHI.10.1.15\n",
      "%T SHRIMP: solving collision and out of vocabulary problems in mobile\n",
      "predictive input with motion gesture\n",
      "%S EPIC #FAIL\n",
      "%A Wang, Jingtao\n",
      "%A Zhai, Shumin\n",
      "%A Canny, John\n",
      "%B Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems\n",
      "%D 2010-04-10\n",
      "%V 1\n",
      "%P 15-24\n",
      "%K camera phones, dictionary-based disambiguation, gestures, mobile devices,\n",
      "mobile phones, multitap, predictive input, t9, text input\n",
      "%* (c) Copyright 2010 ACM\n",
      "%W http://doi.acm.org/10.1145/1753326.1753330\n",
      "%X Dictionary-based disambiguation (DBD) is a very popular solution for text\n",
      "entry on mobile phone keypads but suffers from two problems: 1. the resolution\n",
      "of encoding collision (two or more words sharing the same numeric key sequence)\n",
      "and 2. entering out-of-vocabulary (OOV) words. In this paper, we present\n",
      "SHRIMP, a system and method that addresses these two problems by integrating\n",
      "DBD with camera based motion sensing that enables the user to express\n",
      "preference through a tilting or movement gesture. SHRIMP (Small Handheld Rapid\n",
      "Input with Motion and Prediction) runs on camera phones equipped with a\n",
      "standard 12-key keypad. SHRIMP maintains the speed advantage of DBD driven\n",
      "predictive text input while enabling the user to overcome DBD collision and OOV\n",
      "problems seamlessly without even a mode switch. An initial empirical study\n",
      "demonstrates that SHRIMP can be learned very quickly, performed immediately\n",
      "faster than MultiTap and handled OOV words more efficiently than DBD.\n",
      "\n",
      "%M C.CHI.10.1.25\n",
      "%T Reactive information foraging for evolving goals\n",
      "%S Exploratory search\n",
      "%A Lawrance, Joseph\n",
      "%A Burnett, Margaret\n",
      "%A Bellamy, Rachel\n",
      "%A Bogart, Christopher\n",
      "%A Swart, Calvin\n",
      "%B Proceedings of ACM CHI 2010 Conference on Human Factors in Computing Systems\n",
      "%D 2010-04-10\n",
      "%V 1\n",
      "%P 25-34\n",
      "%K field study, information foraging theory, programming\n",
      "%* (c) Copyright 2010 ACM\n",
      "%W http://doi.acm.org/10.1145/1753326.1753332\n",
      "%X Information foraging models have predicted the navigation paths of people\n",
      "browsing the web and (more recently) of programmers while debugging, but these\n",
      "models do not explicitly model users' goals evolving over time. We present a\n",
      "new information foraging model called PFIS2 that does model information seeking\n",
      "with potentially evolving goals. We then evalua\n"
     ]
    }
   ],
   "source": [
    "ex_path = files['CHI10-1.bib']\n",
    "ex_file = requests.get(f'{hcibib_root}{ex_path}')\n",
    "print(ex_file.text[:5000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These files have line prefixes that indicate different fields. Fields can continue across multiple lines. A blank line separates records.  So we need to parse this.\n",
    "\n",
    "We're going to write a function that processes a file to do exactly that. It is going to be a Python *generator*, so it can be used in a loop but doesn't build a whole list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "_c_re = re.compile(r'^%([A-Z*]) (.*)')\n",
    "_blank_re = re.compile(r'^\\s*$')\n",
    "_bib_codes = {\n",
    "    'T': 'title',\n",
    "    'X': 'abstract',\n",
    "    'A': 'authors',\n",
    "    'D': 'date',\n",
    "    'M': 'id'\n",
    "}\n",
    "def parse_bib(text):\n",
    "    bibrec = {}\n",
    "    last_fld = None\n",
    "    for line in text.splitlines():\n",
    "        cm = _c_re.match(line)\n",
    "        if _blank_re.match(line):\n",
    "            # end of record, emit\n",
    "            if bibrec:\n",
    "                yield bibrec\n",
    "            bibrec = {}\n",
    "        elif cm:\n",
    "            # new field\n",
    "            code = cm.group(1)\n",
    "            value = cm.group(2)\n",
    "            fld = _bib_codes.get(code, None)\n",
    "            if fld:\n",
    "                if fld in bibrec:\n",
    "                    bibrec[fld] += '; ' + value\n",
    "                else:\n",
    "                    bibrec[fld] = value\n",
    "            last_fld = fld\n",
    "        elif last_fld:\n",
    "            # text, add to field\n",
    "            bibrec[last_fld] += ' ' + line\n",
    "            \n",
    "    # if we have an in-progress record, emit it\n",
    "    if bibrec:\n",
    "        yield bibrec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'C.CHI.10.1.1',\n",
       " 'title': 'Estimating residual error rate in recognized handwritten documents using artificial error injection',\n",
       " 'authors': 'Lank, Edward; Stedman, Ryan; Terry, Michael',\n",
       " 'date': '2010-04-10',\n",
       " 'abstract': 'Both handwriting recognition systems and their users are error prone. Handwriting recognizers make recognition errors, and users may miss those errors when verifying output. As a result, it is common for recognized documents to contain residual errors. Unfortunately, in some application domains (e.g. health informatics), tolerance for residual errors in recognized handwriting may be very low, and a desire might exist to maximize user accuracy during verification. In this paper, we present a technique that allows us to measure the performance of a user verifying recognizer output. We inject artificial errors into a set of recognized handwritten forms and show that the rate of injected errors and recognition errors caught is highly correlated in real time. Systems supporting user verification can make use of this measure of user accuracy in a variety of ways. For example, they can force users to slow down or can highlight injected errors that were missed, thus encouraging users to take more care.'}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ex_recs = list(parse_bib(ex_file.text))\n",
    "ex_recs[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a file record extracted!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extracting CHI Papers\n",
    "\n",
    "Now we want to get *all the CHI papers*.\n",
    "\n",
    "Let's define a regex to match a CHI paper file name, `CHI` followed by at least a digit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "chi_re = re.compile(r'^CHI\\d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And get the CHI files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['CHI16-2.bib',\n",
       " 'CHI16-1.bib',\n",
       " 'CHI15-1.bib',\n",
       " 'CHI15-2.bib',\n",
       " 'CHI02-1.bib',\n",
       " 'CHI14-1.bib',\n",
       " 'CHI14-2.bib',\n",
       " 'CHI04-2.bib',\n",
       " 'CHI05-2.bib',\n",
       " 'CHI07-2.bib',\n",
       " 'CHI11-1.bib',\n",
       " 'CHI13-1.bib',\n",
       " 'CHI13-2.bib',\n",
       " 'CHI12-1.bib',\n",
       " 'CHI10-1.bib',\n",
       " 'CHI83.bib',\n",
       " 'CHI12-2.bib',\n",
       " 'CHI11-2.bib',\n",
       " 'CHI82.bib',\n",
       " 'CHI85.bib',\n",
       " 'CHI86.bib',\n",
       " 'CHI87.bib',\n",
       " 'CHI88.bib',\n",
       " 'CHI89.bib',\n",
       " 'CHI90.bib',\n",
       " 'CHI92a.bib',\n",
       " 'CHI92b.bib',\n",
       " 'CHI93a.bib',\n",
       " 'CHI95-2b.bib',\n",
       " 'CHI95-2c.bib',\n",
       " 'CHI96-2b.bib',\n",
       " 'CHI05-1.bib',\n",
       " 'CHI06-1.bib',\n",
       " 'CHI07-1.bib',\n",
       " 'CHI08-1.bib',\n",
       " 'CHI09-1.bib',\n",
       " 'CHI99-2.bib',\n",
       " 'CHI00-1.bib',\n",
       " 'CHI00-2.bib',\n",
       " 'CHI01-1.bib',\n",
       " 'CHI01-2.bib',\n",
       " 'CHI02-2.bib',\n",
       " 'CHI03-1.bib',\n",
       " 'CHI03-2.bib',\n",
       " 'CHI04-1.bib',\n",
       " 'CHI06-2.bib',\n",
       " 'CHI08-2.bib',\n",
       " 'CHI09-2.bib',\n",
       " 'CHI10-2.bib',\n",
       " 'CHI81.bib',\n",
       " 'CHI91.bib',\n",
       " 'CHI92X.bib',\n",
       " 'CHI92Y.bib',\n",
       " 'CHI93X.bib',\n",
       " 'CHI93Y.bib',\n",
       " 'CHI93b.bib',\n",
       " 'CHI94-1.bib',\n",
       " 'CHI94-2a.bib',\n",
       " 'CHI94-2b.bib',\n",
       " 'CHI94-2c.bib',\n",
       " 'CHI94-2d.bib',\n",
       " 'CHI94-2e.bib',\n",
       " 'CHI95-1.bib',\n",
       " 'CHI95-2a.bib',\n",
       " 'CHI96-1.bib',\n",
       " 'CHI96-2a.bib',\n",
       " 'CHI96-2c.bib',\n",
       " 'CHI97-1.bib',\n",
       " 'CHI97-2a.bib',\n",
       " 'CHI97-2b.bib',\n",
       " 'CHI97-2c.bib',\n",
       " 'CHI98-1.bib',\n",
       " 'CHI98-2a.bib',\n",
       " 'CHI98-2b.bib',\n",
       " 'CHI98-2c.bib',\n",
       " 'CHI98-2d.bib',\n",
       " 'CHI99-1.bib']"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi_files = [k for k in files.keys() if chi_re.match(k)]\n",
    "chi_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're going to a list of all the records in all the CHI files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e9bd5fc6662b43439a675cdcae46e3c5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=77.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "13422"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chi_records = []\n",
    "for fk in tqdm(chi_files):\n",
    "    path = files[fk]\n",
    "    data = requests.get(f'{hcibib_root}{path}')\n",
    "    for rec in parse_bib(data.text):\n",
    "        chi_records.append(rec)\n",
    "len(chi_records)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we're going to turn that all into a Pandas data series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>authors</th>\n",
       "      <th>date</th>\n",
       "      <th>abstract</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C.CHI.16.2.1</td>\n",
       "      <td>TactileVR: Integrating Physical Toys into Lear...</td>\n",
       "      <td>Amores, Judith; Benavides, Xavier; Shapira, Lior</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>We present TactileVR, an immersive presence an...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C.CHI.16.2.2</td>\n",
       "      <td>PsychicVR: Increasing mindfulness by using Vir...</td>\n",
       "      <td>Amores, Judith; Benavides, Xavier; Maes, Pattie</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>We present PsychicVR, a proof-of-concept syste...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C.CHI.16.2.3</td>\n",
       "      <td>Haptic Retargeting Video Showcase: Dynamic Rep...</td>\n",
       "      <td>Azmandian, Mahdi; Hancock, Mark; Benko, Hrvoje...</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>Manipulating a virtual object with appropriate...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C.CHI.16.2.4</td>\n",
       "      <td>Reality Editor</td>\n",
       "      <td>Heun, Valentin; Stern-Rodriguez, Eva; Teyssier...</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>The Reality Editor is a tool for empowering a ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C.CHI.16.2.5</td>\n",
       "      <td>Access: A Mobile Application to Improve Access...</td>\n",
       "      <td>Yang, Yi; Hu, Yunqi; Hong, Yidi; Joshi, Varun;...</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>This video introduces Access, a mobile applica...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13417</th>\n",
       "      <td>C.CHI.99.1.576</td>\n",
       "      <td>Mutual Disambiguation of Recognition Errors in...</td>\n",
       "      <td>Oviatt, Sharon</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>As a new generation of multimodal/media system...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13418</th>\n",
       "      <td>C.CHI.99.1.584</td>\n",
       "      <td>Model-Based and Empirical Evaluation of Multim...</td>\n",
       "      <td>Suhm, Bernhard; Waibel, Alex; Myers, Brad</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>Our research addresses the problem of error co...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13419</th>\n",
       "      <td>C.CHI.99.1.592</td>\n",
       "      <td>Cooperative Inquiry: Developing New Technologi...</td>\n",
       "      <td>Druin, Allison</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>In today's homes and schools, children are eme...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13420</th>\n",
       "      <td>C.CHI.99.1.600</td>\n",
       "      <td>Projected Realities: Conceptual Design for Cul...</td>\n",
       "      <td>Gaver, William; Dunne, Anthony</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>As a part of a European Union sponsored projec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13421</th>\n",
       "      <td>C.CHI.99.1.608</td>\n",
       "      <td>Customer-Focused Design Data in a Large, Multi...</td>\n",
       "      <td>Curtis, Paula; Heiserman, Tammy; Jobusch, Davi...</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>Qualitative user-centered design processes suc...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13422 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   id                                              title  \\\n",
       "0        C.CHI.16.2.1  TactileVR: Integrating Physical Toys into Lear...   \n",
       "1        C.CHI.16.2.2  PsychicVR: Increasing mindfulness by using Vir...   \n",
       "2        C.CHI.16.2.3  Haptic Retargeting Video Showcase: Dynamic Rep...   \n",
       "3        C.CHI.16.2.4                                     Reality Editor   \n",
       "4        C.CHI.16.2.5  Access: A Mobile Application to Improve Access...   \n",
       "...               ...                                                ...   \n",
       "13417  C.CHI.99.1.576  Mutual Disambiguation of Recognition Errors in...   \n",
       "13418  C.CHI.99.1.584  Model-Based and Empirical Evaluation of Multim...   \n",
       "13419  C.CHI.99.1.592  Cooperative Inquiry: Developing New Technologi...   \n",
       "13420  C.CHI.99.1.600  Projected Realities: Conceptual Design for Cul...   \n",
       "13421  C.CHI.99.1.608  Customer-Focused Design Data in a Large, Multi...   \n",
       "\n",
       "                                                 authors        date  \\\n",
       "0       Amores, Judith; Benavides, Xavier; Shapira, Lior  2016-05-07   \n",
       "1        Amores, Judith; Benavides, Xavier; Maes, Pattie  2016-05-07   \n",
       "2      Azmandian, Mahdi; Hancock, Mark; Benko, Hrvoje...  2016-05-07   \n",
       "3      Heun, Valentin; Stern-Rodriguez, Eva; Teyssier...  2016-05-07   \n",
       "4      Yang, Yi; Hu, Yunqi; Hong, Yidi; Joshi, Varun;...  2016-05-07   \n",
       "...                                                  ...         ...   \n",
       "13417                                     Oviatt, Sharon  1999-05-15   \n",
       "13418          Suhm, Bernhard; Waibel, Alex; Myers, Brad  1999-05-15   \n",
       "13419                                     Druin, Allison  1999-05-15   \n",
       "13420                     Gaver, William; Dunne, Anthony  1999-05-15   \n",
       "13421  Curtis, Paula; Heiserman, Tammy; Jobusch, Davi...  1999-05-15   \n",
       "\n",
       "                                                abstract  \n",
       "0      We present TactileVR, an immersive presence an...  \n",
       "1      We present PsychicVR, a proof-of-concept syste...  \n",
       "2      Manipulating a virtual object with appropriate...  \n",
       "3      The Reality Editor is a tool for empowering a ...  \n",
       "4      This video introduces Access, a mobile applica...  \n",
       "...                                                  ...  \n",
       "13417  As a new generation of multimodal/media system...  \n",
       "13418  Our research addresses the problem of error co...  \n",
       "13419  In today's homes and schools, children are eme...  \n",
       "13420  As a part of a European Union sponsored projec...  \n",
       "13421  Qualitative user-centered design processes suc...  \n",
       "\n",
       "[13422 rows x 5 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "papers = pd.DataFrame.from_records(chi_records)\n",
    "papers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next step, we need to extract the years from the dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>authors</th>\n",
       "      <th>date</th>\n",
       "      <th>abstract</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>C.CHI.16.2.1</td>\n",
       "      <td>TactileVR: Integrating Physical Toys into Lear...</td>\n",
       "      <td>Amores, Judith; Benavides, Xavier; Shapira, Lior</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>We present TactileVR, an immersive presence an...</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>C.CHI.16.2.2</td>\n",
       "      <td>PsychicVR: Increasing mindfulness by using Vir...</td>\n",
       "      <td>Amores, Judith; Benavides, Xavier; Maes, Pattie</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>We present PsychicVR, a proof-of-concept syste...</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>C.CHI.16.2.3</td>\n",
       "      <td>Haptic Retargeting Video Showcase: Dynamic Rep...</td>\n",
       "      <td>Azmandian, Mahdi; Hancock, Mark; Benko, Hrvoje...</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>Manipulating a virtual object with appropriate...</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C.CHI.16.2.4</td>\n",
       "      <td>Reality Editor</td>\n",
       "      <td>Heun, Valentin; Stern-Rodriguez, Eva; Teyssier...</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>The Reality Editor is a tool for empowering a ...</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>C.CHI.16.2.5</td>\n",
       "      <td>Access: A Mobile Application to Improve Access...</td>\n",
       "      <td>Yang, Yi; Hu, Yunqi; Hong, Yidi; Joshi, Varun;...</td>\n",
       "      <td>2016-05-07</td>\n",
       "      <td>This video introduces Access, a mobile applica...</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13417</th>\n",
       "      <td>C.CHI.99.1.576</td>\n",
       "      <td>Mutual Disambiguation of Recognition Errors in...</td>\n",
       "      <td>Oviatt, Sharon</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>As a new generation of multimodal/media system...</td>\n",
       "      <td>1999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13418</th>\n",
       "      <td>C.CHI.99.1.584</td>\n",
       "      <td>Model-Based and Empirical Evaluation of Multim...</td>\n",
       "      <td>Suhm, Bernhard; Waibel, Alex; Myers, Brad</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>Our research addresses the problem of error co...</td>\n",
       "      <td>1999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13419</th>\n",
       "      <td>C.CHI.99.1.592</td>\n",
       "      <td>Cooperative Inquiry: Developing New Technologi...</td>\n",
       "      <td>Druin, Allison</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>In today's homes and schools, children are eme...</td>\n",
       "      <td>1999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13420</th>\n",
       "      <td>C.CHI.99.1.600</td>\n",
       "      <td>Projected Realities: Conceptual Design for Cul...</td>\n",
       "      <td>Gaver, William; Dunne, Anthony</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>As a part of a European Union sponsored projec...</td>\n",
       "      <td>1999</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13421</th>\n",
       "      <td>C.CHI.99.1.608</td>\n",
       "      <td>Customer-Focused Design Data in a Large, Multi...</td>\n",
       "      <td>Curtis, Paula; Heiserman, Tammy; Jobusch, Davi...</td>\n",
       "      <td>1999-05-15</td>\n",
       "      <td>Qualitative user-centered design processes suc...</td>\n",
       "      <td>1999</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13422 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                   id                                              title  \\\n",
       "0        C.CHI.16.2.1  TactileVR: Integrating Physical Toys into Lear...   \n",
       "1        C.CHI.16.2.2  PsychicVR: Increasing mindfulness by using Vir...   \n",
       "2        C.CHI.16.2.3  Haptic Retargeting Video Showcase: Dynamic Rep...   \n",
       "3        C.CHI.16.2.4                                     Reality Editor   \n",
       "4        C.CHI.16.2.5  Access: A Mobile Application to Improve Access...   \n",
       "...               ...                                                ...   \n",
       "13417  C.CHI.99.1.576  Mutual Disambiguation of Recognition Errors in...   \n",
       "13418  C.CHI.99.1.584  Model-Based and Empirical Evaluation of Multim...   \n",
       "13419  C.CHI.99.1.592  Cooperative Inquiry: Developing New Technologi...   \n",
       "13420  C.CHI.99.1.600  Projected Realities: Conceptual Design for Cul...   \n",
       "13421  C.CHI.99.1.608  Customer-Focused Design Data in a Large, Multi...   \n",
       "\n",
       "                                                 authors        date  \\\n",
       "0       Amores, Judith; Benavides, Xavier; Shapira, Lior  2016-05-07   \n",
       "1        Amores, Judith; Benavides, Xavier; Maes, Pattie  2016-05-07   \n",
       "2      Azmandian, Mahdi; Hancock, Mark; Benko, Hrvoje...  2016-05-07   \n",
       "3      Heun, Valentin; Stern-Rodriguez, Eva; Teyssier...  2016-05-07   \n",
       "4      Yang, Yi; Hu, Yunqi; Hong, Yidi; Joshi, Varun;...  2016-05-07   \n",
       "...                                                  ...         ...   \n",
       "13417                                     Oviatt, Sharon  1999-05-15   \n",
       "13418          Suhm, Bernhard; Waibel, Alex; Myers, Brad  1999-05-15   \n",
       "13419                                     Druin, Allison  1999-05-15   \n",
       "13420                     Gaver, William; Dunne, Anthony  1999-05-15   \n",
       "13421  Curtis, Paula; Heiserman, Tammy; Jobusch, Davi...  1999-05-15   \n",
       "\n",
       "                                                abstract  year  \n",
       "0      We present TactileVR, an immersive presence an...  2016  \n",
       "1      We present PsychicVR, a proof-of-concept syste...  2016  \n",
       "2      Manipulating a virtual object with appropriate...  2016  \n",
       "3      The Reality Editor is a tool for empowering a ...  2016  \n",
       "4      This video introduces Access, a mobile applica...  2016  \n",
       "...                                                  ...   ...  \n",
       "13417  As a new generation of multimodal/media system...  1999  \n",
       "13418  Our research addresses the problem of error co...  1999  \n",
       "13419  In today's homes and schools, children are eme...  1999  \n",
       "13420  As a part of a European Union sponsored projec...  1999  \n",
       "13421  Qualitative user-centered design processes suc...  1999  \n",
       "\n",
       "[13422 rows x 6 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "papers['year'] = papers['date'].str.replace(r'^(\\d{4}).*', r'\\1').astype('i4')\n",
    "papers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can save this data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "papers.to_csv('chi-papers.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}