hn-posting-hours-pakistan.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Finding Five Hot Hacker News Posting Hours in Pakistan\n",
    "\n",
    "In this project, we'll aim to find 5 hot Hacker News posting hours for people in Pakistan so they can have a higher chance of receiving more comments on their posts. Hacker News is a popular website where technology related posts or stories are voted and commented on. We are interested in posts who's titles begin with `Ask HN` or `Show HN`. Users submit `Ask HN` posts to ask the Hacker News community a specific question.Likewise, users submit `Show HN` posts to show the Hacker News community a project, product, or just generally something interesting.\n",
    "\n",
    "We'll analyze existing data about Hacker News posts to find the hot HN posting hours in Pakistan. To support our recommendation, we'll try to find out:\n",
    "\n",
    "- Do Ask HN or Show HN receive more comments on average.\n",
    "- Do posts created at a certain time receive more comments on average."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary of Results\n",
    "\n",
    "After analyzing the data, the conclusion we reached is that the hottest hour for creating posts in Pakistan is `3:00pm`with higher chance of receiving more comments. The other top four hours that guarantee receiving comments are `2:00am`, `8:00pm`, `4:00pm`and `9:00pm`.\n",
    "\n",
    "For more details, please refer to the full analysis below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring Existing Data\n",
    "\n",
    "To avoid spending money on organizing a survey, we'll try to make use of existing data to determine whether we can reach any reliable result. \n",
    "\n",
    "The dataset is publicly available on the Kaggle site with full detail of the columns, but it has been reduced from almost 300,000 rows to approximately 20,000 rows by removing all submissions that did not receive any comments. Below, is the quick exploration of data, available to me offline, stored in `hacker_news.csv`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['12224879',\n",
       "  'Interactive Dynamic Video',\n",
       "  'http://www.interactivedynamicvideo.com/',\n",
       "  '386',\n",
       "  '52',\n",
       "  'ne0phyte',\n",
       "  '8/4/2016 11:52'],\n",
       " ['10975351',\n",
       "  'How to Use Open Source and Shut the Fuck Up at the Same Time',\n",
       "  'http://hueniverse.com/2016/01/26/how-to-use-open-source-and-shut-the-fuck-up-at-the-same-time/',\n",
       "  '39',\n",
       "  '10',\n",
       "  'josep2',\n",
       "  '1/26/2016 19:30'],\n",
       " ['11964716',\n",
       "  \"Florida DJs May Face Felony for April Fools' Water Joke\",\n",
       "  'http://www.thewire.com/entertainment/2013/04/florida-djs-april-fools-water-joke/63798/',\n",
       "  '2',\n",
       "  '1',\n",
       "  'vezycash',\n",
       "  '6/23/2016 22:20'],\n",
       " ['11919867',\n",
       "  'Technology ventures: From Idea to Enterprise',\n",
       "  'https://www.amazon.com/Technology-Ventures-Enterprise-Thomas-Byers/dp/0073523429',\n",
       "  '3',\n",
       "  '1',\n",
       "  'hswarna',\n",
       "  '6/17/2016 0:01'],\n",
       " ['10301696',\n",
       "  'Note by Note: The Making of Steinway L1037 (2007)',\n",
       "  'http://www.nytimes.com/2007/11/07/movies/07stein.html?_r=0',\n",
       "  '8',\n",
       "  '2',\n",
       "  'walterbell',\n",
       "  '9/30/2015 4:12']]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Read the data\n",
    "import csv\n",
    "with open('hacker_news.csv') as csvfile:\n",
    "    hn = list(csv.reader(csvfile, delimiter=','))\n",
    "\n",
    "# Extracting headers\n",
    "headers = hn[0]\n",
    "# Removing headers from data\n",
    "hn = hn[1:]\n",
    "# Display first five rows\n",
    "hn[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Extracting Ask HN and Show HN Posts\n",
    "As we mentioned in the introduction that we need to find the posting hours with higher chance of receiving comments and only `Ask HN` and `Show HN` posts receive comments. For the purpose of our analysis, we need to extract posts that starts with either Ask HN or Show HN.\n",
    "\n",
    "The dataset provides information about the Hacker News posts. Every post in the dataset contains a `title` column that specifies the title of the post. In order to extract `Ask HN` and `Show HN` posts, we need to filter the posts whose title begin with either \"Ask HN\" or \"Show HN\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Ask Posts:  1744\n",
      "Total Show Posts:  1162\n",
      "Total Other Posts:  17194\n"
     ]
    }
   ],
   "source": [
    "# Extracting posts that starts with Ask HN or Show HN\n",
    "import re\n",
    "\n",
    "ask_posts = []\n",
    "show_posts = []\n",
    "other_posts = []\n",
    "\n",
    "for row in hn:\n",
    "    title = row[1]\n",
    "    if re.search(\"^Ask HN\",title,re.IGNORECASE):\n",
    "        ask_posts.append(row)\n",
    "    elif re.search(\"^Show HN\",title,re.IGNORECASE):\n",
    "        show_posts.append(row)\n",
    "    else:\n",
    "        other_posts.append(row)\n",
    "\n",
    "print(\"Total Ask Posts: \",len(ask_posts))\n",
    "print(\"Total Show Posts: \",len(show_posts))\n",
    "print(\"Total Other Posts: \",len(other_posts))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last code block, we separated posts that begin with `Ask HN` and `Show HN` into two list of lists named `ask_posts` and `show_posts` and the remaining posts in `other_posts` list of lists. From a quick scan, it looks like:\n",
    "\n",
    "- The number of Ask HN posts are more than the Show HN posts. \n",
    "- Both combined are quite less than the other posts.\n",
    "\n",
    "We can infer that the site is used more for asking technical questions from the HN Community rather than displaying projects, product or any other interesting stuff.\n",
    "\n",
    "Below are the first five rows in the `ask_posts` list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['12296411',\n",
       "  'Ask HN: How to improve my personal website?',\n",
       "  '',\n",
       "  '2',\n",
       "  '6',\n",
       "  'ahmedbaracat',\n",
       "  '8/16/2016 9:55'],\n",
       " ['10610020',\n",
       "  'Ask HN: Am I the only one outraged by Twitter shutting down share counts?',\n",
       "  '',\n",
       "  '28',\n",
       "  '29',\n",
       "  'tkfx',\n",
       "  '11/22/2015 13:43'],\n",
       " ['11610310',\n",
       "  'Ask HN: Aby recent changes to CSS that broke mobile?',\n",
       "  '',\n",
       "  '1',\n",
       "  '1',\n",
       "  'polskibus',\n",
       "  '5/2/2016 10:14'],\n",
       " ['12210105',\n",
       "  'Ask HN: Looking for Employee #3 How do I do it?',\n",
       "  '',\n",
       "  '1',\n",
       "  '3',\n",
       "  'sph130',\n",
       "  '8/2/2016 14:20'],\n",
       " ['10394168',\n",
       "  'Ask HN: Someone offered to buy my browser extension from me. What now?',\n",
       "  '',\n",
       "  '28',\n",
       "  '17',\n",
       "  'roykolak',\n",
       "  '10/15/2015 16:38']]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display first five Ask HN posts\n",
    "ask_posts[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below are the first five rows in the `show_posts` list of lists:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['10627194',\n",
       "  'Show HN: Wio Link  ESP8266 Based Web of Things Hardware Development Platform',\n",
       "  'https://iot.seeed.cc',\n",
       "  '26',\n",
       "  '22',\n",
       "  'kfihihc',\n",
       "  '11/25/2015 14:03'],\n",
       " ['10646440',\n",
       "  'Show HN: Something pointless I made',\n",
       "  'http://dn.ht/picklecat/',\n",
       "  '747',\n",
       "  '102',\n",
       "  'dhotson',\n",
       "  '11/29/2015 22:46'],\n",
       " ['11590768',\n",
       "  'Show HN: Shanhu.io, a programming playground powered by e8vm',\n",
       "  'https://shanhu.io',\n",
       "  '1',\n",
       "  '1',\n",
       "  'h8liu',\n",
       "  '4/28/2016 18:05'],\n",
       " ['12178806',\n",
       "  'Show HN: Webscope  Easy way for web developers to communicate with Clients',\n",
       "  'http://webscopeapp.com',\n",
       "  '3',\n",
       "  '3',\n",
       "  'fastbrick',\n",
       "  '7/28/2016 7:11'],\n",
       " ['10872799',\n",
       "  'Show HN: GeoScreenshot  Easily test Geo-IP based web pages',\n",
       "  'https://www.geoscreenshot.com/',\n",
       "  '1',\n",
       "  '9',\n",
       "  'kpsychwave',\n",
       "  '1/9/2016 20:45']]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Display first five Show HN posts\n",
    "show_posts[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Determine Ask HN or Show HN Receive More Comments\n",
    "\n",
    "Let's begin with finding out whether the **Ask HN** posts receive more comments on average or the **Show HN** posts. This should be a good start for finding out the hot posting hours in Pakistan that guarantee receiving comments.\n",
    "\n",
    "The dataset provides information about the number of comments each post has received over a year. We will examine the column `num_comments` which contains the total number of comments against every post. In order to determine which posts receive more comments, we'll calculate the **Average Ask Comments** and **Average Show Comments**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average Ask Comments:  14.038417431192661\n",
      "Average Show Comments:  10.31669535283993\n"
     ]
    }
   ],
   "source": [
    "# Determine if Ask HN or Show HN posts receive more comments\n",
    "ask_comments = [int(row[4]) for row in ask_posts]\n",
    "show_comments = [int(row[4]) for row in show_posts]\n",
    "\n",
    "avg_ask_comments = sum(ask_comments)/len(ask_comments)\n",
    "avg_show_comments = sum(show_comments)/len(show_comments)\n",
    "\n",
    "print(\"Average Ask Comments: \",avg_ask_comments)\n",
    "print(\"Average Show Comments: \",avg_show_comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to the analysis above, we determined that on average, `Ask HN` posts receive more comments than `Show HN` posts. Since `Ask HN` posts are more likely to receive comments, we'll focus our remaining analysis just on these posts."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ask HN Posts & Comments by Hour Created\n",
    "\n",
    "Now that we have determined which posts receive more comments on average. It's time to address the second question in the introduction i.e. if posts created at a certain time are more likely to attract comments. This take us another step closer to finding the hotest posting hours in Pakistan.\n",
    "\n",
    "We'll use the following steps to perform this analysis:\n",
    "\n",
    "1. Calculate the amount of Ask HN posts created in each hour of the day, along with the number of comments received.\n",
    "2. Calculate the average number of comments Ask HN posts receive by hour created.\n",
    "\n",
    "In the next code block, we will tackle the first step — calculating the amount of `Ask HN` posts and comments by hour created. We will calculate the number of posts created at each hour and the total number of comments received each hour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Amount of Ask HN posts and comments by hour created\n",
    "import datetime as dt\n",
    "\n",
    "created_date = [row[6] for row in ask_posts]\n",
    "\n",
    "counts_by_hour = {}\n",
    "comments_by_hour = {}\n",
    "\n",
    "result_list = list(zip(created_date,ask_comments))\n",
    "\n",
    "for date,comment in result_list:\n",
    "    hour = dt.datetime.strptime(date,\"%m/%d/%Y %H:%M\").strftime(\"%H\")\n",
    "    if hour not in counts_by_hour:\n",
    "        counts_by_hour[hour] = 1\n",
    "        comments_by_hour[hour] = comment\n",
    "    elif hour in counts_by_hour:\n",
    "        counts_by_hour[hour] += 1\n",
    "        comments_by_hour[hour] += comment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have created two dictionaries:\n",
    "\n",
    "- `counts_by_hour`: contains the number of **Ask HN** posts created during each hour of the day.\n",
    "- `comments_by_hour`: contains the corresponding number of comments **Ask HN** posts created at each hour received.\n",
    "\n",
    "We'll use these two dictionaries to calculate the average number of comments for posts created during each hour of the day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['09', 5.5777777777777775],\n",
       " ['13', 14.741176470588234],\n",
       " ['10', 13.440677966101696],\n",
       " ['14', 13.233644859813085],\n",
       " ['16', 16.796296296296298],\n",
       " ['23', 7.985294117647059],\n",
       " ['12', 9.41095890410959],\n",
       " ['17', 11.46],\n",
       " ['15', 38.5948275862069],\n",
       " ['21', 16.009174311926607],\n",
       " ['20', 21.525],\n",
       " ['02', 23.810344827586206],\n",
       " ['18', 13.20183486238532],\n",
       " ['03', 7.796296296296297],\n",
       " ['05', 10.08695652173913],\n",
       " ['19', 10.8],\n",
       " ['01', 11.383333333333333],\n",
       " ['22', 6.746478873239437],\n",
       " ['08', 10.25],\n",
       " ['04', 7.170212765957447],\n",
       " ['00', 8.127272727272727],\n",
       " ['06', 9.022727272727273],\n",
       " ['07', 7.852941176470588],\n",
       " ['11', 11.051724137931034]]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Calculate avg number of comments for Ask HN posts by hour\n",
    "avg_by_hour = [[hour, comments_by_hour[hour]/counts_by_hour[hour]] for hour in counts_by_hour]\n",
    "avg_by_hour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the last code block, we calculated the average number of comments for posts created during each hour of the day and stored the results in a list of lists named `avg_by_hour`.\n",
    "\n",
    "Now that we have the number of comments received against each hour of the day, it is difficult to identify the hours that received more comments than the others. To find the top 5 hotest hours we need to sort `avg_by_hour` in the order of decreasing comments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Top 5 Hours for 'Ask HN' Comments\n",
      "15:00 38.59 average comments per post\n",
      "02:00 23.81 average comments per post\n",
      "20:00 21.52 average comments per post\n",
      "16:00 16.80 average comments per post\n",
      "21:00 16.01 average comments per post\n"
     ]
    }
   ],
   "source": [
    "# Sort by Comments\n",
    "avg_by_hour = sorted(avg_by_hour,key=lambda x:x[1], reverse=True)\n",
    "\n",
    "# Format printing\n",
    "print(\"Top 5 Hours for 'Ask HN' Comments\")\n",
    "for hour,average in avg_by_hour[0:5]:\n",
    "    print(\"{} {:.2f} average comments per post\".format(dt.datetime.strptime(hour,\"%H\").strftime(\"%H:00\"),average))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "After analyzing the data, the conclusion we reached is that the hottest hour for creating posts in Pakistan is `3:00pm` with higher chance of receiving more comments. The other top four hours that guarantee receiving comments are `2:00am`, `8:00pm`, `4:00pm` and `9:00pm`."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}