preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to /Users/krut/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /Users/krut/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from utility import clean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing train data\n",
    "df_train = pd.read_csv('data/train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Before preprocessing:  '\"\\nFair use rationale for Image:Wonju.jpg\\n\\nThanks for uploading Image:Wonju.jpg. I notice the image page specifies that the image is being used under fair use but there is no explanation or rationale as to why its use in Wikipedia articles constitutes fair use. In addition to the boilerplate fair use template, you must also write out on the image description page a specific explanation or rationale for why using this image in each article is consistent with fair use.\\n\\nPlease go to the image description page and edit it to include a fair use rationale.\\n\\nIf you have uploaded other fair use media, consider checking that you have specified the fair use rationale on those pages too. You can find a list of \\'image\\' pages you have edited by clicking on the \"\"my contributions\"\" link (it is located at the very top of any Wikipedia page when you are logged in), and then selecting \"\"Image\"\" from the dropdown box. Note that any fair use images uploaded after 4 May, 2006, and lacking such an explanation will be deleted one week after they have been uploaded, as described on criteria for speedy deletion. If you have any questions please ask them at the Media copyright questions page. Thank you. (talk • contribs • ) \\nUnspecified source for Image:Wonju.jpg\\n\\nThanks for uploading Image:Wonju.jpg. I noticed that the file\\'s description page currently doesn\\'t specify who created the content, so the copyright status is unclear. If you did not create this file yourself, then you will need to specify the owner of the copyright. If you obtained it from a website, then a link to the website from which it was taken, together with a restatement of that website\\'s terms of use of its content, is usually sufficient information. However, if the copyright holder is different from the website\\'s publisher, then their copyright should also be acknowledged.\\n\\nAs well as adding the source, please add a proper copyright licensing tag if the file doesn\\'t have one already. If you created/took the picture, audio, or video then the  tag can be used to release it under the GFDL. If you believe the media meets the criteria at Wikipedia:Fair use, use a tag such as  or one of the other tags listed at Wikipedia:Image copyright tags#Fair use. See Wikipedia:Image copyright tags for the full list of copyright tags that you can use.\\n\\nIf you have uploaded other files, consider checking that you have specified their source and tagged them, too. You can find a list of files you have uploaded by following [ this link]. Unsourced and untagged images may be deleted one week after they have been tagged, as described on criteria for speedy deletion. If the image is copyrighted under a non-free license (per Wikipedia:Fair use) then the image will be deleted 48 hours after . If you have any questions please ask them at the Media copyright questions page. Thank you. (talk • contribs • ) \"'\n",
      "\n",
      "After preprocessing:  ' fair use rationale image wonju jpg thank upload image wonju jpg notice image page specify image use fair use explanation rationale use wikipedia article constitute fair use addition boilerplate fair use template must also write image description page specific explanation rationale use image article consistent fair use please go image description page edit include fair use rationale upload fair use media consider check specify fair use rationale page find list image page edit click contributions link locate top wikipedia page log select image dropdown box note fair use image upload 4 may 2006 lack explanation delete one week upload describe criteria speedy deletion question please ask media copyright question page thank talk contribs unspecified source image wonju jpg thank upload image wonju jpg notice file s description page currently specify create content copyright status unclear create file need specify owner copyright obtain website link website take together restatement website s term use content usually sufficient information however copyright holder different website s publisher copyright also acknowledge well add source please add proper copyright license tag file one already create take picture audio video tag use release gfdl believe media meet criteria wikipedia fair use use tag one tag list wikipedia image copyright tag fair use see wikipedia image copyright tag full list copyright tag use upload file consider check specify source tag find list file upload follow link unsourced untagged image may delete one week tag describe criteria speedy deletion image copyright non free license per wikipedia fair use image delete 48 hours question please ask media copyright question page thank talk contribs '\n"
     ]
    }
   ],
   "source": [
    "# Before preprocessing\n",
    "text = df_train['comment_text'].iloc[10]\n",
    "print(\"Before preprocessing: \", repr(text))\n",
    "\n",
    "# After preprocessing\n",
    "print(\"\\nAfter preprocessing: \", repr(clean(text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}