Smarter Data Collection for Microsoft Civic Graph

In an effort to make Civic Graph a little bit smarter, I developed four building blocks to help improve the quality (i.e. accuracy, completeness) of the data stored as well as automate aspects of the data collection process. They are:

• Web Scrapers
• Analytics Toolbox
• Classifier
• [Pipeline](/References/Pipeline Diagram.pdf) to integrate scrapers, classifier, and analysis with existing Civic Graph

View the full write-up [here](/References/Handoff for Briana.pdf).

---

Additional Resources:

I've compiled a list of tools and resources that I used throughout the project. They cover a range of topics including:

• Web Scraping
• Data Analysis with Python
• Text mining
• Natural Language Processing
• Machine Learning

Libraries, tools, APIs:

• BeautifulSoup

• spaCy

• textacy

• nltk

• scikit-learn

Concepts:

• Supervised Learning

• Support Vector Machine

• Named-entity Recognition (NER)

• Term Frequency-Inverse Document Frequency (tf-idf)

• Feature Extraction

Books:

• Python for Data Analysis

• Natural Language Processing with Python

• Web Scraping with Python

Academic Papers:

• Processing Named Entities in Text

---

Created by Hannah Cutler during my internship at