MovieLens 1M ETL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  ml-1m.zip\n",
      "   creating: ml-1m/\n",
      "  inflating: ml-1m/movies.dat        \n",
      "  inflating: ml-1m/ratings.dat       \n",
      "  inflating: ml-1m/README            \n",
      "  inflating: ml-1m/users.dat         \n"
     ]
    }
   ],
   "source": [
    "! wget -q http://files.grouplens.org/datasets/movielens/ml-1m.zip\n",
    "! unzip ml-1m.zip"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define constants\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "BASE_DIR = '.' # Modify this if needed to the local directory that the MovieLens 1M Dataset has been unzipped into. \n",
    "MOVIELENS_DIR = BASE_DIR + '/ml-1m/'\n",
    "USER_DATA_FILE = 'users.dat'\n",
    "MOVIE_DATA_FILE = 'movies.dat'\n",
    "RATING_DATA_FILE = 'ratings.dat'\n",
    "AGES = { 1: \"Under 18\", 18: \"18-24\", 25: \"25-34\", 35: \"35-44\", 45: \"45-49\", 50: \"50-55\", 56: \"56+\" }\n",
    "OCCUPATIONS = { 0: \"other or not specified\", 1: \"academic/educator\", 2: \"artist\", 3: \"clerical/admin\",\n",
    "                4: \"college/grad student\", 5: \"customer service\", 6: \"doctor/health care\",\n",
    "                7: \"executive/managerial\", 8: \"farmer\", 9: \"homemaker\", 10: \"K-12 student\", 11: \"lawyer\",\n",
    "                12: \"programmer\", 13: \"retired\", 14: \"sales/marketing\", 15: \"scientist\", 16: \"self-employed\",\n",
    "                17: \"technician/engineer\", 18: \"tradesman/craftsman\", 19: \"unemployed\", 20: \"writer\" }\n",
    "RATINGS_CSV_FILE = 'ml1m_ratings.csv'\n",
    "USERS_CSV_FILE = 'ml1m_users.csv'\n",
    "MOVIES_CSV_FILE = 'ml1m_movies.csv'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load MovieLens 1M data\n",
    "\n",
    "The MovieLens 1M Dataset can be downloaded from http://files.grouplens.org/datasets/movielens/ml-1m.zip. We load the data about users, movies and ratings into dataframes. \n",
    "\n",
    "A few quirks in this dataset:\n",
    "\n",
    "* The ids for users and movies are 1-based,\n",
    "* not all movies have descriptions, and \n",
    "* not all movies are rated. \n",
    "\n",
    "To make it easy to simply use series from the ratings dataframe as training inputs and output to the Keras model, we do the following:\n",
    "\n",
    "* We set max_userid to the maximum user id in the ratings,\n",
    "* we set max_movieid to the maximum movie id in the ratings, and\n",
    "* we add columns (user_emb_id and movie_emb_id) whose values are the user or movie ids minus one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1000209 ratings loaded\n",
      "Saved to ml1m_ratings.csv\n"
     ]
    }
   ],
   "source": [
    "ratings = pd.read_csv(os.path.join(MOVIELENS_DIR, RATING_DATA_FILE), \n",
    "                    sep='::', \n",
    "                    engine='python', \n",
    "                    encoding='latin-1',\n",
    "                    names=['userid', 'movieid', 'rating', 'timestamp'])\n",
    "max_userid = ratings['userid'].drop_duplicates().max()\n",
    "max_movieid = ratings['movieid'].drop_duplicates().max()\n",
    "ratings['user_emb_id'] = ratings['userid'] - 1\n",
    "ratings['movie_emb_id'] = ratings['movieid'] - 1\n",
    "print len(ratings), 'ratings loaded'\n",
    "ratings.to_csv(RATINGS_CSV_FILE, \n",
    "               sep='\\t', \n",
    "               header=True, \n",
    "               encoding='latin-1', \n",
    "               columns=['userid', 'movieid', 'rating', 'timestamp', 'user_emb_id', 'movie_emb_id'])\n",
    "print 'Saved to', RATINGS_CSV_FILE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6040 descriptions of 6040 users loaded.\n",
      "Saved to ml1m_users.csv\n"
     ]
    }
   ],
   "source": [
    "users = pd.read_csv(os.path.join(MOVIELENS_DIR, USER_DATA_FILE), \n",
    "                    sep='::', \n",
    "                    engine='python', \n",
    "                    encoding='latin-1',\n",
    "                    names=['userid', 'gender', 'age', 'occupation', 'zipcode'])\n",
    "users['age_desc'] = users['age'].apply(lambda x: AGES[x])\n",
    "users['occ_desc'] = users['occupation'].apply(lambda x: OCCUPATIONS[x])\n",
    "print len(users), 'descriptions of', max_userid, 'users loaded.'\n",
    "users.to_csv(USERS_CSV_FILE, \n",
    "             sep='\\t', \n",
    "             header=True, \n",
    "             encoding='latin-1',\n",
    "             columns=['userid', 'gender', 'age', 'occupation', 'zipcode', 'age_desc', 'occ_desc'])\n",
    "print 'Saved to', USERS_CSV_FILE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3883 descriptions of 3952 movies loaded.\n",
      "Saved to ml1m_movies.csv\n"
     ]
    }
   ],
   "source": [
    "movies = pd.read_csv(os.path.join(MOVIELENS_DIR, MOVIE_DATA_FILE), \n",
    "                    sep='::', \n",
    "                    engine='python', \n",
    "                    encoding='latin-1',\n",
    "                    names=['movieid', 'title', 'genre'])\n",
    "print len(movies), 'descriptions of', max_movieid, 'movies loaded.'\n",
    "movies.to_csv(MOVIES_CSV_FILE, \n",
    "              sep='\\t', \n",
    "              header=True, \n",
    "              columns=['movieid', 'title', 'genre'])\n",
    "print 'Saved to', MOVIES_CSV_FILE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6040 of the 6040 users rate at least one movie.\n"
     ]
    }
   ],
   "source": [
    "print len(ratings['userid'].drop_duplicates()), 'of the', max_userid, 'users rate at least one movie.'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "3706 of the 3952 movies are rated.\n"
     ]
    }
   ],
   "source": [
    "print len(ratings['movieid'].drop_duplicates()), 'of the', max_movieid, 'movies are rated.'"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}