05_Reddit_Pushshift_API.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Reddit with the pushshift API\n",
    "This notebook give you examples of how to use the pushshift API for querying Reddit data.\n",
    "\n",
    "* Pushshift doc:  https://github.com/pushshift/api\n",
    "* FAQ about Pushshift: https://www.reddit.com/r/pushshift/comments/bcxguf/new_to_pushshift_read_this_faq/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define a convenient function to get data from Pushshift:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_pushshift_data(data_type, params):\n",
    "    \"\"\"\n",
    "    Gets data from the pushshift api.\n",
    " \n",
    "    data_type can be 'comment' or 'submission'\n",
    "    The rest of the args are interpreted as payload.\n",
    " \n",
    "    Read more: https://github.com/pushshift/api\n",
    "    \n",
    "    This function is inspired from:\n",
    "    https://www.jcchouinard.com/how-to-use-reddit-api-with-python/\n",
    "    \"\"\"\n",
    " \n",
    "    base_url = f\"https://api.pushshift.io/reddit/search/{data_type}/\"\n",
    "    request = requests.get(base_url, params=params)\n",
    "    print('Query:')\n",
    "    print(request.url)\n",
    "    try: \n",
    "        data = request.json().get(\"data\")\n",
    "    except:\n",
    "        print('--- Request failed ---')\n",
    "        data = []\n",
    "    return data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function accepts the parameters of the pushshift API detailed in the doc at https://github.com/pushshift/api. An example is given below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example of request to the API\n",
    "Let us collect the comments written in the last 2 day in the subreddit `askscience`. The number of results returned is limited to 100, the upper limit of the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# parameters for the pushshift API\n",
    "data_type = \"comment\"    # accept \"comment\" or \"submission\", search in comments or submissions\n",
    "params = {\n",
    "    \"subreddit\" : \"askscience\", # limit to one or a list of subreddit(s)\n",
    "    \"after\" : \"7d\", # Select the timeframe. Epoch value or Integer + \"s,m,h,d\" (i.e. \"second\", \"minute\", \"hour\", \"day\")\n",
    "    \"size\" : 100, # Number of results to return (limited to max 100 in the API)\n",
    "    \"author\" : \"![deleted]\" # limit to a list of authors or ignore authors with a \"!\" mark in front\n",
    "}\n",
    "# Note: the option \"aggs\" (aggregate) has been de-activated in the API\n",
    "\n",
    "data = get_pushshift_data(data_type, params)\n",
    "if data: # control if something is returned\n",
    "    df = pd.DataFrame.from_records(data)\n",
    "    print('Some of the data returned:')\n",
    "    df[['author', 'subreddit', 'score', 'created_utc', 'body']].head()\n",
    "else:\n",
    "    print('The returned data is empty. Change the parameters.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authors of comments\n",
    "Let us collect the authors of comments in a subreddit during the last days. The next function helps bypassing the limit of results by sending queries multiple times, avoiding collecting duplicate authors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the list of unique authors of comments in the API results\n",
    "# bypass the limit of 100 results by sending multiple queries\n",
    "def get_unique_authors(n_results, params):\n",
    "    results_per_request = 100 # default nb of results per query\n",
    "    n_queries = n_results // results_per_request + 1\n",
    "    author_list = []\n",
    "    author_neg_list = [\"![deleted]\"]\n",
    "    for query in range(n_queries):\n",
    "        params[\"author\"] = author_neg_list\n",
    "        data = get_pushshift_data(data_type=\"comment\", params=params)\n",
    "        df = pd.DataFrame.from_records(data)\n",
    "        if df.empty:\n",
    "            return author_list\n",
    "        authors = list(df['author'].unique())\n",
    "        # add ! mark\n",
    "        authors_neg = [\"!\"+ a for a in authors]\n",
    "        author_list += authors\n",
    "        author_neg_list += authors_neg\n",
    "    return author_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us make a list of authors commenting on the subreddit \"askscience\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Ask for the authors of comments in the last days, colect at least \"n_results\"\n",
    "subreddit = \"askscience\"\n",
    "data_type = \"comment\"\n",
    "params = {\n",
    "    \"subreddit\" : subreddit,\n",
    "    \"after\" : \"2d\"\n",
    "}\n",
    "n_results = 500\n",
    "author_list = get_unique_authors(n_results, params)\n",
    "print(\"Number of authors:\",len(author_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the list of authors obtained, let us collect where else the commented posts (other subreddits)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Collect the subreddits where the authors wrote comments and the number of comments\n",
    "from collections import Counter\n",
    "data_type = \"comment\"\n",
    "params = {\n",
    "    \"size\" : 100\n",
    "}\n",
    "subreddits_count = Counter()\n",
    "for author in author_list:\n",
    "    params[\"author\"] = author\n",
    "    print(params[\"author\"])\n",
    "    data = get_pushshift_data(data_type=data_type, params=params)\n",
    "    if data: # in case the resquest failed and data is empty\n",
    "        df = pd.DataFrame.from_records(data)\n",
    "        subreddits_count += Counter(dict(df['subreddit'].value_counts()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network of subreddits (ego-graph)\n",
    "Let us build the ego-graph of the subreddit. Other subreddits will be connected to the main one if the users commented in the other subreddits as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# module for networks\n",
    "import networkx as nx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 0.05\n",
    "G = nx.Graph()\n",
    "G.add_node(subreddit)\n",
    "self_refs = subreddits_count[subreddit]\n",
    "for sub,value in subreddits_count.items():\n",
    "    post_ratio = value/self_refs\n",
    "    if post_ratio >= threshold:\n",
    "        G.add_edge(subreddit,sub, weight=post_ratio)\n",
    "print(\"Total number of edges in the graph:\",G.number_of_edges())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is an alternative way of generating the graph using pandas dataframes instead of a for loop (it might scale better on bigger graphs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "threshold = 0.05\n",
    "subreddits_count_df = pd.DataFrame.from_dict(subreddits_count, orient='index', columns=['total'])\n",
    "subreddits_ratio_df = subreddits_count_df/subreddits_count_df.loc[subreddit]\n",
    "subreddits_ratio_df.rename(columns={'total': 'weight'}, inplace=True)\n",
    "filtered_sr_df = subreddits_ratio_df[subreddits_ratio_df['weight'] >= threshold].copy() # filter weights < threshold\n",
    "filtered_sr_df['source'] = subreddit\n",
    "filtered_sr_df['target'] = filtered_sr_df.index\n",
    "Gdf = nx.from_pandas_edgelist(filtered_sr_df, source='source', target='target', edge_attr=True)\n",
    "print(\"Total number of edges in the graph:\",Gdf.number_of_edges())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the graph to a file\n",
    "path = 'egograph.gexf'\n",
    "nx.write_gexf(G,path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Network of subreddit neighbors\n",
    "This second collection makes a distinction between the related subreddits. For each author, all the subreddits where he/she commented will be connected together. The weight of each connection will be proportional to the number of users commenting in both subreddits joined by the connection. The ego-graph becomes an approximate neighbor network for the central subreddit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data_type = \"comment\"\n",
    "params = {\n",
    "    \"size\" : 100\n",
    "}\n",
    "count_list = []\n",
    "for author in author_list:\n",
    "    params[\"author\"] = author\n",
    "    print(params[\"author\"])\n",
    "    data = get_pushshift_data(data_type=data_type, params=params)\n",
    "    if data:\n",
    "        df = pd.DataFrame.from_records(data)\n",
    "        count_list.append(Counter(dict(df['subreddit'].value_counts())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "threshold = 0.05\n",
    "G = nx.Graph()\n",
    "\n",
    "for author_sub_count in count_list:\n",
    "    sub_list = author_sub_count.most_common(10)\n",
    "    # Compute all the combinations of subreddit pairs\n",
    "    sub_combinations = list(itertools.combinations(sub_list, 2))\n",
    "    for sub_pair in sub_combinations:\n",
    "        node1 = sub_pair[0][0]\n",
    "        node2 = sub_pair[1][0]\n",
    "        if G.has_edge(node1, node2):\n",
    "            G[node1][node2]['weight'] +=1\n",
    "        else:\n",
    "            G.add_edge(node1, node2, weight=1)\n",
    "print(\"Total number of edges {}, and nodes {}\".format(G.number_of_edges(),G.number_of_nodes()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Sparsify the graph\n",
    "to_remove = [edge for edge in G.edges.data() if edge[2]['weight'] < 2]\n",
    "G.remove_edges_from(to_remove)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove isolated nodes\n",
    "G.remove_nodes_from(list(nx.isolates(G)))\n",
    "print(\"Total number of edges {}, and nodes {}\".format(G.number_of_edges(),G.number_of_nodes()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write the graph to a file\n",
    "path = 'graph.gexf'\n",
    "nx.write_gexf(G,path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An example of the graph visualization you can obtain using Gephi:\n",
    "![Reddit neighbors](figures/redditneighbors.png \"Reddit neighbors\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}