Recommender Code.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recommender Code\n",
    "\n",
    "The classic recommender tutorial uses the [movielens data set](https://grouplens.org/datasets/movielens/). It is similar to using the iris or MNIST data set for other algorithms. Let's do a code along to get an idea of how this all works!\n",
    "\n",
    "\n",
    "Looking for more datasets? Check out: https://gist.github.com/entaroadun/1653794"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.sql import SparkSession"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "spark = SparkSession.builder.appName('rec').getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With Collaborative filtering we make predictions (filtering) about the interests of a user by collecting preferences or taste information from many users (collaborating). The underlying assumption is that if a user A has the same opinion as a user B on an issue, A is more likely to have B's opinion on a different issue x than to have the opinion on x of a user chosen randomly.\n",
    "\n",
    "The image below (from Wikipedia) shows an example of collaborative filtering. At first, people rate different items (like videos, images, games). Then, the system makes predictions about a user's rating for an item not rated yet. The new predictions are built upon the existing ratings of other users with similar ratings with the active user. In the image, the system predicts that the user will not like the video.\n",
    "\n",
    "<img src=https://upload.wikimedia.org/wikipedia/commons/5/52/Collaborative_filtering.gif />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Spark MLlib library for Machine Learning provides a Collaborative Filtering implementation by using Alternating Least Squares. The implementation in MLlib has these parameters:\n",
    "\n",
    "* numBlocks is the number of blocks used to parallelize computation (set to -1 to auto-configure).\n",
    "* rank is the number of latent factors in the model.\n",
    "* iterations is the number of iterations to run.\n",
    "* lambda specifies the regularization parameter in ALS.\n",
    "* implicitPrefs specifies whether to use the explicit feedback ALS variant or one adapted for implicit feedback data.\n",
    "* alpha is a parameter applicable to the implicit feedback variant of ALS that governs the baseline confidence in preference observations.\n",
    "\n",
    "Let's see this all in action!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark.ml.evaluation import RegressionEvaluator\n",
    "from pyspark.ml.recommendation import ALS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = spark.read.csv('movielens_ratings.csv',inferSchema=True,header=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Row(movieId=2, rating=3.0, userId=0)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------------------+------------------+------------------+\n",
      "|summary|           movieId|            rating|            userId|\n",
      "+-------+------------------+------------------+------------------+\n",
      "|  count|              1501|              1501|              1501|\n",
      "|   mean| 49.40572951365756|1.7741505662891406|14.383744170552964|\n",
      "| stddev|28.937034065088994| 1.187276166124803| 8.591040424293272|\n",
      "|    min|                 0|               1.0|                 0|\n",
      "|    max|                99|               5.0|                29|\n",
      "+-------+------------------+------------------+------------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "data.describe().show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can do a split to evaluate how well our model performed, but keep in mind that it is very hard to know conclusively how well a recommender system is truly working for some topics. Especially if subjectivity is involved, for example not everyone that loves star wars is going to love star trek, even though a recommendation system may suggest otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Smaller dataset so we will use 0.8 / 0.2\n",
    "(training, test) = data.randomSplit([0.8, 0.2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the recommendation model using ALS on the training data\n",
    "als = ALS(maxIter=5, regParam=0.01, userCol=\"userId\", itemCol=\"movieId\", ratingCol=\"rating\")\n",
    "model = als.fit(training)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see hwo the model performed!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Evaluate the model by computing the RMSE on the test data\n",
    "predictions = model.transform(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+------+-----------+\n",
      "|movieId|rating|userId| prediction|\n",
      "+-------+------+------+-----------+\n",
      "|     31|   1.0|    27| 0.91522497|\n",
      "|     31|   4.0|    12|  0.2069124|\n",
      "|     31|   1.0|     4|   2.301581|\n",
      "|     31|   2.0|    25|  0.8806035|\n",
      "|     85|   3.0|     1|  0.6333409|\n",
      "|     85|   3.0|     6|  0.8793151|\n",
      "|     85|   1.0|    15| -2.2477286|\n",
      "|     85|   1.0|     4| -1.5924109|\n",
      "|     85|   4.0|     7|  0.5359794|\n",
      "|     65|   1.0|    16|-0.13987958|\n",
      "|     65|   2.0|     3| 0.08086155|\n",
      "|     65|   2.0|    15|  2.9638734|\n",
      "|     53|   5.0|    21|  3.5669563|\n",
      "|     78|   1.0|    12| 0.49899876|\n",
      "|     78|   1.0|    19|    1.36678|\n",
      "|     78|   1.0|     4| 0.38657337|\n",
      "|     78|   1.0|     2| 0.06448199|\n",
      "|     34|   3.0|    25|   1.484213|\n",
      "|     34|   1.0|    14|  1.5834944|\n",
      "|     34|   4.0|     2|  0.3634969|\n",
      "+-------+------+------+-----------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "predictions.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Root-mean-square error = 1.756887166750377\n"
     ]
    }
   ],
   "source": [
    "evaluator = RegressionEvaluator(metricName=\"rmse\", labelCol=\"rating\",predictionCol=\"prediction\")\n",
    "rmse = evaluator.evaluate(predictions)\n",
    "print(\"Root-mean-square error = \" + str(rmse))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The RMSE described our error in terms of the stars rating column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now that we have the model, how would you actually supply a recommendation to a user?\n",
    "\n",
    "The same way we did with the test data! For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "single_user = test.filter(test['userId']==11).select(['movieId','userId'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+\n",
      "|movieId|userId|\n",
      "+-------+------+\n",
      "|      0|    11|\n",
      "|     11|    11|\n",
      "|     16|    11|\n",
      "|     19|    11|\n",
      "|     20|    11|\n",
      "|     36|    11|\n",
      "|     38|    11|\n",
      "|     43|    11|\n",
      "|     50|    11|\n",
      "|     72|    11|\n",
      "|     75|    11|\n",
      "|     94|    11|\n",
      "+-------+------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# User had 10 ratings in the test data set \n",
    "# Realistically this should be some sort of hold out set!\n",
    "single_user.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "reccomendations = model.transform(single_user)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------+------+----------+\n",
      "|movieId|userId|prediction|\n",
      "+-------+------+----------+\n",
      "|     20|    11| 2.6914935|\n",
      "|     94|    11| 1.9483502|\n",
      "|     50|    11|  1.924752|\n",
      "|     38|    11| 1.8383508|\n",
      "|     19|    11|0.97057307|\n",
      "|     36|    11|0.73484004|\n",
      "|     75|    11|0.65968406|\n",
      "|      0|    11|0.36861914|\n",
      "|     11|    11|0.12782133|\n",
      "|     16|    11|-0.9434327|\n",
      "|     72|    11|-1.0054189|\n",
      "|     43|    11|-1.2666295|\n",
      "+-------+------+----------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "reccomendations.orderBy('prediction',ascending=False).show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}