0b_data_prep_arxiv.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "85bac51e",
   "metadata": {},
   "source": [
    "# Part 0 - Data preparation\n",
    "\n",
    "In this notebook we will download the arXiv dataset and save it to S3. We will also do some light data preprocessing by only keeping the columns we need, filtering out reviews that are too short, and limiting the size of the datasets.\n",
    "\n",
    "To read more, please check out https://towardsdatascience.com/setting-up-a-text-summarisation-project-introduction-526622eea4a8."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "880efb0c",
   "metadata": {},
   "source": [
    "First of all we want to make sure that the relevent libraries are installed on this machine:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2fe74630",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install \"sagemaker>=2.48.0\" \"transformers==4.6.1\" \"datasets[s3]==1.6.2\" --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe89716a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install rouge_score"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74713c30",
   "metadata": {},
   "source": [
    "We will download the dataset directly from the Kaggle website so we need to install the Kaggle Python package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a9749dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install kaggle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92cd6652",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['KAGGLE_USERNAME'] = \"<your-kaggle-username>\"\n",
    "os.environ['KAGGLE_KEY'] = \"<your-kaggle-api-key>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "413c346b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import kaggle\n",
    "kaggle.api.authenticate()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a4d7f1cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "kaggle.api.dataset_download_files('Cornell-University/arxiv', path=\".\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d2b814a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!unzip arxiv.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fd20d8c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "beba5f6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mv arxiv.zip raw_data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8100b134",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mv arxiv-metadata-oai-snapshot.json raw_data/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6dc9f150",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "dataset = load_dataset(\"arxiv_dataset\", data_dir='./raw_data/', split='train', ignore_verifications=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "605aae99",
   "metadata": {},
   "source": [
    "The original dataset is too long, so we shuffle it and limit the number of records to 25,000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f6efedf",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = dataset.shuffle(seed=42)\n",
    "dataset = dataset.select(range(25000))\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "62c0f4ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.DataFrame(dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "95d41258",
   "metadata": {},
   "outputs": [],
   "source": [
    " # only keep columns that are required\n",
    "df = df[['abstract', 'title']]\n",
    "df = df.rename(columns={\"abstract\": \"text\", \"title\": \"summary\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45044aa0",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.replace(r'\\n',' ', regex=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "657c2475",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.display.max_colwidth = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0149280b",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6c4750",
   "metadata": {},
   "source": [
    "## Filtering the dataset\n",
    "\n",
    "We want to discard reviews and titles that are too short, so that our model can produce more interesting summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53102dfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "cutoff_summary = 5\n",
    "cutoff_text = 20\n",
    "df = df[(df['summary'].apply(lambda x: len(x.split()) >= cutoff_summary)) & (df['text'].apply(lambda x: len(x.split()) >= cutoff_text))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e0b156d",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0769ad53",
   "metadata": {},
   "source": [
    "## Limiting the size of the datasets and splitting\n",
    "\n",
    "We want to limit the size of the datasets so that training of the model can finish in a reasonable amount of time. This is a decision that we might want to revisit in the experimentation phase if we want to increase the performance of the model. We then split the dataset into test (80%), validation (10%), and test (10%)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5f42ced",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.sample(20000, random_state=43)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a29b5891",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# split the dataset into train, val, and test\n",
    "df_train, df_val, df_test = np.split(df.sample(frac=1, random_state=44), [int(0.8*len(df)), int((0.9)*len(df))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4c36f75",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train.to_csv('data/train.csv', index=False)\n",
    "df_val.to_csv('data/val.csv', index=False)\n",
    "df_test.to_csv('data/test.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98223a6c",
   "metadata": {},
   "source": [
    "## Save the data as CSV files and upload them to S3\n",
    "\n",
    "We need to upload the data to S3 in order to train the model at a later point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54c83b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "sess = sagemaker.Session()\n",
    "bucket = sess.default_bucket()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ccac070",
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp data/train.csv s3://$bucket/summarization/data/train.csv\n",
    "!aws s3 cp data/val.csv s3://$bucket/summarization/data/val.csv\n",
    "!aws s3 cp data/test.csv s3://$bucket/summarization/data/test.csv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_latest_p37",
   "language": "python",
   "name": "conda_pytorch_latest_p37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}