project3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2342813f",
   "metadata": {},
   "source": [
    "## Student Name: \n",
    "## Student Email:"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "84e6ab65",
   "metadata": {},
   "source": [
    "# Project 3: The Smart City Slicker\n",
    "\n",
    "Imagine you are a stakeholder in a rising Smart City and want to know more about themes and concepts about existing smart cities. You also want to know where does your smart city place among others. In this project, you will perform \n",
    "exploratory data analysis, often shortened to EDA, to examine a data from the [2015 Smart City Challenge](https://www.transportation.gov/smartcity) to find facts about the data and communicating those facts through text analysis and visualizations.\n",
    "\n",
    "In order to explore the data and visualize it, some modifications might need to be made to the data along the way. This is often referred to as data preprocessing or cleaning.\n",
    "Though data preprocessing is technically different from EDA, EDA often exposes problems with the data that need to be fixed in order to continue exploring.\n",
    "Because of this tight coupling, you have to clean the data as necessary to help understand the data.\n",
    "\n",
    "In this project, you will apply your knowledge about data cleaning, machine learning, visualizations, and databases to explore smart city applications.\n",
    "\n",
    "**Part 1** of the notebook will explore and clean the data. \\\n",
    "**Part 2** will take the results of the preprocessed data to create models and visualizations.\n",
    "\n",
    "Empty cells are code cells. \n",
    "Cells denoted with [Your Answer Here] are markdown cells.\n",
    "Edit and add as many cells as needed."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6e8dcba1",
   "metadata": {},
   "source": [
    "Output file for this notebook is shown as a table for display purposes. Note: The city name can be Norman, OK or OK Norman.\n",
    "\n",
    "| city | raw text | clean text | clusterid | topicids | \n",
    "| -- | -- | -- | -- | -- | \n",
    "|Norman, OK | Test, test , and testing. | test test test | 0 | T1, T2| "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89fd47ce",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "The Dataset: 2015 Smart City Challenge Applicants (non-finalist).\n",
    "In this project you will use the applicant's PDFs as a dataset.\n",
    "The dataset is from the U.S Department of Transportation Smart City Challenge.\n",
    "\n",
    "On the website page for the data, you can find some basic information about the challenge. This is an interesting dataset. Think of the questions that you might be able to answer! A few could be:\n",
    "\n",
    "1. Can I identify frequently occurring words that could be removed during data preprocessing?\n",
    "2. Where are the applicants from?\n",
    "3. Are there multiple entries for the same city in different applicantions?\n",
    "4. What are the major themes and concepts from the smart city applicants?\n",
    "\n",
    "Let's load the data!"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7aace966",
   "metadata": {},
   "source": [
    "## Loading and Handling files (Required)\n",
    "\n",
    "Load data from `smartcity/`. \n",
    "\n",
    "To extract the data from the pdf files, use the [pypdf.pdf.PdfFileReader](https://pypdf.readthedocs.io/en/stable/index.html) class.\n",
    "It will allow you to extract pages and pdf files and add them to a data structure (dataframe, list, dictionary, etc).\n",
    "To install the module, use the command `pipenv install pypdf`.\n",
    "You only need to handle PDF files, handling docx is not necessary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef0afb30",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d8ed6e32",
   "metadata": {},
   "source": [
    "Create a data structure to add the city name and raw text. You can choose to split the city name from the file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e4905f1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5019a8c3",
   "metadata": {},
   "source": [
    "## Cleaning Up PDFs (Required)\n",
    "\n",
    "One of the more frustrating aspects of PDF is loading the data into a readable format. The first order of business will be to preprocess the data. To start, you can use code provided by Text Analytics with Python, [Chapter 3](https://github.com/dipanjanS/text-analytics-with-python/blob/master/New-Second-Edition/Ch03%20-%20Processing%20and%20Understanding%20Text/Ch03a%20-%20Text%20Wrangling.ipynb): [contractions.py](https://github.com/dipanjanS/text-analytics-with-python/blob/master/New-Second-Edition/Ch05%20-%20Text%20Classification/contractions.py) (Pages 136-137), and [text_normalizer.py](https://github.com/dipanjanS/text-analytics-with-python/blob/master/New-Second-Edition/Ch05%20-%20Text%20Classification/text_normalizer.py) (Pages 155-156). Feel free to download the scripts or add the code directly to the notebook (please note this code is performed on dataframes).\n",
    "\n",
    "In addition to the data cleaning provided by the textbook, you will need to:\n",
    "1. Consider removing terms that may effect clustering and topic modeling. Words to consider are cities, states, common words (smart, city, page, etc.). Keep in mind n-gram combinations are important; this can also be revisited later depending on your model's performance.\n",
    "2. Check the data to remove applicants that text was not processed correctly. Do not remove more than 15 cities from the data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8142e498",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1473a3e3",
   "metadata": {},
   "source": [
    "#### Add the cleaned text to the structure you created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3737fe02",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "82cc947b",
   "metadata": {},
   "source": [
    "### Clean Up: Discussion\n",
    "Answer the questions below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f1ba98d",
   "metadata": {},
   "source": [
    "#### Which Smart City applicants did you remove? What issues did you see with the documents?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffebf5a5",
   "metadata": {},
   "source": [
    "[Your Answer Here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1620ed74",
   "metadata": {},
   "source": [
    "#### Explain what additional text processing methods you used and why."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae42fc81",
   "metadata": {},
   "source": [
    "[Your Answer Here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15817355",
   "metadata": {},
   "source": [
    "#### Did you identify any potientally problematic words?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0ad6082",
   "metadata": {},
   "source": [
    "[Your Answer Here]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a1507fbe",
   "metadata": {},
   "source": [
    "## Experimenting with Clustering Models (Required)\n",
    "\n",
    "Now, you'll start to explore models to find the optimal clustering model. In this section, you'll explore [K-means](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html), [Hierarchical](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html), and [DBSCAN](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html#sklearn.cluster.DBSCAN) clustering algorithms.\n",
    "Create these algorithms with k_clusters for K-means and Hierarchical.\n",
    "For each cell in the table provide the [Silhouette score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html#sklearn.metrics.silhouette_score), [Calinski and Harabasz score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.calinski_harabasz_score.html#sklearn.metrics.calinski_harabasz_score), and [Davies-Bouldin score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.davies_bouldin_score.html#sklearn.metrics.davies_bouldin_score).\n",
    "\n",
    "In each cell, create an array to store the values.\n",
    "For example, \n",
    "\n",
    "|Algorithm| k = 9 | k = 18| k = 36 | Optimal k| \n",
    "|--|--|--|--|--|\n",
    "|K-means| [S,CH,DB]| [S,CH,DB] | [S,CH,DB] | [S,CH,DB] |\n",
    "|Hierarchical |[S,CH,DB]| [S,CH,DB]| [S,CH,DB] | [S,CH,DB]|\n",
    "|DBSCAN | X | X | X | [S,CH,DB] |\n",
    "\n",
    "\n",
    "\n",
    "### Optimality \n",
    "You will need to find the optimal k for K-means and Hierarchical algorithms.\n",
    "Find the optimality for k in the range 2 to 50.\n",
    "Provide the code used to generate the optimal k and provide justification for your approach.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8da2b033",
   "metadata": {},
   "source": [
    "|Algorithm| k = 9 | k = 18| k = 36 | Optimal k| \n",
    "|--|--|--|--|--|\n",
    "|K-means|--|--|--|--|\n",
    "|Hierarchical |--|--|--|--|\n",
    "|DBSCAN | X | X | X | -- |\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dc8c7ab",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d96d23a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6c20fd69",
   "metadata": {},
   "source": [
    "#### How did you approach finding the optimal k?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "753c54da",
   "metadata": {},
   "source": [
    "[Your answer here]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f79ec635",
   "metadata": {},
   "source": [
    "#### What algorithm do you believe is the best? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3136b0a4",
   "metadata": {},
   "source": [
    "[Your Answer]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "2e45a2a3",
   "metadata": {},
   "source": [
    "### Add Cluster ID to output file\n",
    "In your data structure, add the cluster id for each smart city respectively. Show the to append the clusterid code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ad83e3b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "959e7275",
   "metadata": {},
   "source": [
    "### Save Model\n",
    "\n",
    "After finding the best model, it is desirable to have a way to persist the model for future use without having to retrain. Save the model using [model persistance](https://scikit-learn.org/stable/model_persistence.html). This model should be saved in the same directory as this notebook and should be loaded as the model for your `project3.py`.\n",
    "\n",
    "Save the model as `model.pkl`. You do not have to use pickle, but be sure to save the persistance using one of the methods listed in the link."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c80938e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3fe5a0c9",
   "metadata": {},
   "source": [
    "## Derving Themes and Concepts (Required)\n",
    "\n",
    "Perform Topic Modeling on the cleaned data. Provide the top five words for `TOPIC_NUM = Best_k` as defined in the section above. Feel free to reference [Chapter 6](https://github.com/dipanjanS/text-analytics-with-python/tree/master/New-Second-Edition/Ch06%20-%20Text%20Summarization%20and%20Topic%20Models) for more information on Topic Modeling and Summarization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b684bc24",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "915516dc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a2a3896",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1573fe65",
   "metadata": {},
   "source": [
    "### Extract themes\n",
    "Write a theme for each topic (atleast a sentence each)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9737d8df",
   "metadata": {},
   "source": [
    "[Your Answer]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "21cf24c6",
   "metadata": {},
   "source": [
    "[Your Answer]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "75e191f4",
   "metadata": {},
   "source": [
    "[Your Answer]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94ab7df0",
   "metadata": {},
   "source": [
    "### Add Topid ID to output file\n",
    "Add the top two topics for each smart city to the data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e937841",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e568652",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "39f9c240",
   "metadata": {},
   "source": [
    "## Gathering Applicant Summaries and Keywords (Extra Credit Section)\n",
    "\n",
    "For each smart city applicant, gather a summary and keywords that are important to that document. Gensim is outdated; try a spacy or nltk method.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a86cbe26",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b545ac4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2d27ce37",
   "metadata": {},
   "source": [
    "### Add Summaries and Keywords\n",
    "Add summary and keywords to output file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c09357ff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4f9cb2c4",
   "metadata": {},
   "source": [
    "## Write output data (Required)\n",
    "\n",
    "The output data should be written as a TSV file.\n",
    "You can use `to_csv` method from Pandas for this if you are using a DataFrame.\n",
    "\n",
    "`Syntax: df.to_csv('file.tsv', sep = '')` \\\n",
    "`df.to_csv('smartcity_eda.tsv', sep='\\t')`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58827464",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7a18b8ff",
   "metadata": {},
   "source": [
    "# Moving Forward\n",
    "Now that you have explored the dataset, take the important features and functions to create your `project3.py`.\n",
    "Please refer to the project spec for more guidance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a6675ba",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}