0_data-collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Predicting Airbnb Listing Price | Data Collection\n",
    "\n",
    "The data for this project comes from [Inside Airbnb](http://insideairbnb.com/get-the-data.html), an independent, non-commercial project that collects public data from the travel and accomodations company Airbnb.\n",
    "\n",
    "This notebook will show how I scraped the necessary files from the web and consolidated them into files of listings data and reviews data.\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "import requests\n",
    "import warnings\n",
    "import pandas as pd\n",
    "\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "warnings.filterwarnings(\"ignore\")\n",
    "pd.set_option('display.max_columns', None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Web Scraping\n",
    "\n",
    "Since I used data sourced from many files found on the same webpage, I decided to write a script that scrapes that page for the relevant URLs to my project. \n",
    "\n",
    "To complete this task, I used the requests and BeautifulSoup packages. First, I used requests to capture the response from the url where all of the csv files are hosted: http://insideairbnb.com/get-the-data.html.\n",
    "\n",
    "After catching that response in a variable, I read the text into another variable. I turned the text variable into a BeautifulSoup object using the BeautifulSoup function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'http://insideairbnb.com/get-the-data.html'\n",
    "r = requests.get(url)\n",
    "html_doc = r.text\n",
    "soup = BeautifulSoup(html_doc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating the BeautifulSoup object, I extracted the text from the HTML anchor tags, where the csv links are found in the ‘href’ attribute of the tag. I stored the names of the csv urls in a list. Since the links led to gzipped (.csv.gz) files, I called the list `zipped_links`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "zipped_links = []\n",
    "for link in soup.find_all('a'):\n",
    "    link_url = link.get('href')\n",
    "    if (link_url is not None) and ('listings.csv' in link_url or 'reviews.csv' in link_url) and ('los-angeles' in link_url) and ('visualisations' not in link_url):\n",
    "        zipped_links.append(link_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, I wrote the information from the online csv files to local files. Using a context manager, I looped through `zipped_links` to create custom filenames for each list item and write to that file. The filename includes the city name, the date the data was collected, and the information category (listings or reviews). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def write_files(ls, directory):\n",
    "    '''write information from the links to their own files'''\n",
    "    for link in ls:\n",
    "        file_url_split = link.split('/')\n",
    "        filename = file_url_split[-4] + '_' + file_url_split[-3] + '_' + file_url_split[-1]\n",
    "        # if the file doesn't exist in our directory, write to the file\n",
    "        if(not os.path.isfile(directory + filename)):\n",
    "            with open(directory + filename, \"wb\") as f:\n",
    "                r = requests.get(link)\n",
    "                f.write(r.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ALL FILES WRITTEN!\n"
     ]
    }
   ],
   "source": [
    "directory = '/Users/limesncoconuts2/datasets/airbnb-web/'\n",
    "\n",
    "# remove files that are 0 bytes (program timed out while they were being written previously)\n",
    "for file in os.listdir(directory):\n",
    "    if os.path.getsize(directory + file) == 0:\n",
    "        os.remove(directory + file)\n",
    "\n",
    "# check if all files have been written\n",
    "if len(zipped_links) != len(os.listdir(directory)):\n",
    "    write_files(zipped_links, directory)\n",
    "print('ALL FILES WRITTEN!')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some links to files on the website are broken, so they have to be removed from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove files that are of less than 1kb \n",
    "for file in os.listdir(directory):\n",
    "    if os.path.getsize(directory + file) < 1000:\n",
    "        os.remove(directory + file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining Data\n",
    "\n",
    "After the raw files were scraped, created, and stored, the data was consolidated from monthly files into one single file for each listings and reviews. Because there are  hundreds of thousands of lines of data in each file, it helped to create functions that did the heavy lifting:\n",
    "\n",
    "`consolidate_data` checks if the consolidated csv file for either listings or reviews data has been created for the designated city. If the file has not been created, it runs the `combine_files` function for that city, and then creates the csv file for that city.\n",
    "\n",
    "`combine_files` goes through files in the directory and checks for the designated city files of the specified kind. Then it appends the names of the files of that city to a list, and passes the list and the directory name to the `concat_files` function.\n",
    "\n",
    "`concat_files` creates a pandas dataframe for each file name in the list of files, then appends the dataframe to a list of dataframes. After all files in the list have been converted to pandas dataframes, it concatenates the dataframes together, drops duplicate rows, and resets the dataframe index.\n",
    "\n",
    "`export_csv` checks if the desired csv file does not exist in the current working directory, then converts the dataframe to a csv file and moves the the desired folder in the destination directory.\n",
    "\n",
    "Even though I started by analyzing just Los Angeles, having functions will also allow me to scale my analysis to multiple cities in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def consolidate_data(city, directory, destination):\n",
    "    \"\"\" Checks if the csv file for either listings\n",
    "        or reviews data has been created for the designated\n",
    "        city in the destination folder.\n",
    "        If the file has not been created, run the combine_listings\n",
    "        or combine_reviews function for that city, and then creates\n",
    "        the csv file for that city.\n",
    "    \"\"\"\n",
    "    \n",
    "    filename = city + '_listings.csv'\n",
    "    if(not os.path.isfile(destination + filename)):\n",
    "        listings_df = combine_files(city, directory, 'listings')\n",
    "        export_csv(filename, listings_df, destination)\n",
    "    \n",
    "    filename = city + '_reviews.csv'\n",
    "    if(not os.path.isfile(destination + filename)):\n",
    "        reviews_df = combine_files(city, directory, 'reviews')\n",
    "        export_csv(filename, reviews_df, destination)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def combine_files(city, directory, kind):\n",
    "    \"\"\" Goes through files in the directory and checks for the\n",
    "        designated city files of the specified kind. Appends the names of the\n",
    "        files of that city to a list, and passes the list and the directory \n",
    "        name to the concat_files function.\n",
    "    \"\"\"\n",
    "    target_files = []\n",
    "    \n",
    "    for file in os.listdir(directory):\n",
    "        if city in file and kind in file:\n",
    "            target_files.append(file)\n",
    "    return concat_files(target_files, directory) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def concat_files(file_list, directory):\n",
    "    \"\"\"Creates a pandas dataframe for each file name in the \n",
    "       list of files. Appends the dataframe to a list of dataframes. After all files\n",
    "       in the list have been converted to pandas dataframes,\n",
    "       concatenate the dataframes together, drop duplicates,\n",
    "       and reset the dataframe index.\n",
    "    \"\"\"\n",
    "    all_dfs = []\n",
    "        \n",
    "    for file in file_list:\n",
    "        df = pd.read_csv(directory + file)\n",
    "        all_dfs.append(df)\n",
    "    \n",
    "    concat_all = pd.concat(all_dfs)    \n",
    "    concat_all.drop_duplicates(inplace=True)\n",
    "    concat_all.reset_index(drop=True, inplace=True)\n",
    "        \n",
    "    return concat_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def export_csv(filename, df, destination):\n",
    "    \"\"\" If the desired csv file does not exist in the current\n",
    "        working directory, convert the dataframe to a csv file\n",
    "        and move the the desired folder in the destination directory.\n",
    "    \"\"\"\n",
    "    current_dir = os.getcwd() + '/' + filename\n",
    "    if(not os.path.isfile(current_dir)):\n",
    "        df.to_csv(filename, index=False)\n",
    "        shutil.move(os.path.join(current_dir), os.path.join(destination, filename))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After I created the functions, I defined the directory (the pathway where the raw data is stored) and destination folder (where I wanted to store the concatenated data on the computer) and ran the functions. The outcome was a single listings file and single reviews file for Los Angeles."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "directory = '/Users/limesncoconuts2/datasets/airbnb-web/'\n",
    "destination = '/Users/limesncoconuts2/datasets/airbnb/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "city = 'los-angeles'\n",
    "if(not os.path.isfile(destination + city + '_listings.csv') or not os.path.isfile(destination + city + '_reviews.csv')):\n",
    "    consolidate_data(city, directory, destination)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}