WebScraping-Day1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "3e89c5b3",
   "metadata": {},
   "source": [
    "<img align=\"left\" src=\"https://ithaka-labs.s3.amazonaws.com/static-files/images/tdm/tdmdocs/tapi-logo-small.png\" />\n",
    "\n",
    "This notebook free for educational reuse under [Creative Commons CC BY License](https://creativecommons.org/licenses/by/4.0/).\n",
    "\n",
    "Created by [Elizabeth Wickes](https://ischool.illinois.edu/people/elizabeth-wickeshttps://ischool.illinois.edu/people/elizabeth-wickes) for the 2023 Text Analysis Pedagogy Institute, with support from [Constellate](https://constellate.org).\n",
    "\n",
    "For questions/comments/improvements, email wickes1@illinois.edu.<br />\n",
    "____"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68f932d1",
   "metadata": {},
   "source": [
    "# Web Scraping Toolkit 1\n",
    "\n",
    "This is lesson 1 of 3 in the educational series on `Web Scraping`. This notebook is intended to teach the core problem solving perspectives and tools for webscraping. \n",
    "\n",
    "**Audience:** `Teachers` / `Learners` / `Researchers`\n",
    "\n",
    "**Use case:** `Tutorial` / `How-To` \n",
    "\n",
    "**Difficulty:** Intermediate\n",
    "\n",
    "**Completion time:** 90 minutes\n",
    "\n",
    "**Knowledge Required:** \n",
    "\n",
    "* Python basics (variables, flow control, functions, lists)\n",
    "* Basic file operations (open, close, read, write)\n",
    "\n",
    "**Knowledge Recommended:**\n",
    "\n",
    "* basic html/websites\n",
    "\n",
    "**Learning Objectives:**\n",
    "After this lesson, learners will be able to:\n",
    "\n",
    "1. Make basic determinations about which tool to use for extracting data from a website.\n",
    "\n",
    "**Research Pipeline:**\n",
    "\n",
    "1. You have a research question and data in mind.\n",
    "2. You've found some data you want to use.\n",
    "2. **The data is on a website somewhere and you want to get it off the site and into a data file.**\n",
    "3. You do your analysis or other data prep!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "157c0555",
   "metadata": {},
   "source": [
    "# Required Python Libraries\n",
    "\n",
    "* `requests` for downloading things\n",
    "\n",
    "## Install Required Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8a220f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Install Libraries ###\n",
    "\n",
    "# Using !pip installs\n",
    "!pip install requests\n",
    "\n",
    "# Using %%bash magic with apt-get and yes prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "5480e2a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "### Import Libraries ###\n",
    "\n",
    "#3rd party\n",
    "import requests\n",
    "from lxml import etree\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "import re\n",
    "import time\n",
    "import pathlib\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f53edaa2",
   "metadata": {},
   "source": [
    "# Introduction\n",
    "\n",
    "\n",
    "* purpose of this of our shop is to sort of lay a foundation of what web scraping is\n",
    "* understand the kinds of situations you may find\n",
    "* review the problem solving approaches\n",
    "* understand the commonly used technologies and when to use them\n",
    "\n",
    "Leading into the next workshop, where we focus on exploring xpath and regular expressions. Why are these separate workshops? Those can be pretty heavy to learn and sometimes you really don't need them. Sometimes things like Google sheets can take care of your needs. The whole purpose of this is to explore the breadth of problem solving options available for web scraping so you can pick the one right for what you are working with.\n",
    "\n",
    "## What this is not going to be\n",
    "\n",
    "To make it quick, the text you want needs to be actual text somewhere on the website. \n",
    "\n",
    "This means we aren't going to be talking about:\n",
    "\n",
    "* extracting text or other content from PDFs\n",
    "* OCRing data/etc from images\n",
    "* deep dive into working with APIs\n",
    "\t* although we will talk a bit about APIs\n",
    "\n",
    "## What I love about web scraping\n",
    "Aside from getting data you may not be able to otherwise...\n",
    "\n",
    "There are so many interesting situations that involve some deeply creative problem solving strategies. You have to be a little bit like a private investigator. Sleuthing through the website trying to figure out if there are structures in the pages that we can use to our advantage.\n",
    "\n",
    "## The hierarchy of your time\n",
    "\n",
    "Your time and research time is important. Same for the people you might be teaching these things to. When you're first learning about a lot of programming and other tools sometimes your first thoughts are, hey let me write a script for this! Sure, getting more practice can be good, but you've got to respect your own time and needs.\n",
    "# Where does this come into a project?\n",
    "Generally you should already have some sort of question or ask your learners if they have their research question or area. Web scraping comes in during the phase of data gathering or data discovery.\n",
    "\n",
    "And this workshop sort of presumes that you have data somewhere that you can put your eyeballs on and say, I need that data. And once you got to that point of like, okay, the data exists. How do I get it?\n",
    "\n",
    "## The first question about web scraping is not about web scraping\n",
    "Always check first, does this data exist somewhere in a more accessible format? I've done a lot of scraping for things in the past but now other people have put datasets from those things up.\n",
    "\n",
    "Pro: you can download a thing and it's already data!\n",
    "Con: it may be an older snapshot, may not have all the things you want, etc.\n",
    "\n",
    "However, this may be enough for a small proof of concept. Don't discount something quick and easy just to explore the vibes!\n",
    "\n",
    "## So where can this data be found?\n",
    "\n",
    "Now, this isn't a workshop about how to find data. However, when you are teaching these workshops, this might be a good place to advertise your services etc. Here are some of my recommendations (this is going to be pretty US specific):\n",
    "\n",
    "1. https://www.re3data.org/\n",
    "\t1. Repository of repositories, you may be surprised by what's already out there!\n",
    "2. https://commons.datacite.org/\n",
    "\t1. Search metadata for all DOIs registered under datacite, which does contain a good amount of data! Has cool stuff, but the results can be a bit noisy\n",
    "3. https://www.icpsr.umich.edu/web/pages/ICPSR/index.html\n",
    "\t1. Lots of social science data, amazing search engine\n",
    "4. https://data.gov/ or local city/metro area data repository\n",
    "\t1. Government data etc\n",
    "5. Others that are....less of my favorite\n",
    "\t1. https://www.google.com/publicdata/directory\n",
    "\t2. https://www.kaggle.com/datasets\n",
    "6. Also search if there's an API, and sometimes you need to search for that specifically\n",
    "\n",
    "Other important factors to stress: \n",
    "\n",
    "1. Check your local resources! \n",
    "\t1. Subject librarian or other library resources. Be sure to ask directly if there might be something available. Not everything makes it to the website perfectly.\n",
    "2. Lots and lots of googling\n",
    "\t1. Sometimes you can find stuff in odd places or there are little personal data collections that people may have online. \n",
    "\t2. I like to give myself a good hour or so to dive deep to try and find something, but I usually have to set a timer so I eventually actually stop...\n",
    "\n",
    "So eventually you're to a point where you can put eyeballs on the data in front of you, it's on a website, you can copy/paste it, etc. However, you can't find it as a nice pretty dataset to download. This is where web scraping fits in. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9c98f19",
   "metadata": {},
   "source": [
    "\n",
    "# Briefly, how web pages do the thing\n",
    "\n",
    "(Very briefly, very high level)\n",
    "\n",
    "Web pages use HTML as a markup language to dictate how the content should be displayed. Headers, bold, etc. It also allows for things like hyperlinks, pictures, and other content to be displayed. This markup is also just text in specific formats.\n",
    "\n",
    "Web browsers \"parse\" or read this text and attempt to understand the content versus the structure of the HTML given to it. The web browser then uses it's tools to display it for you. \n",
    "\n",
    "For example, `<b>This will be bold text.</b>` will display that text as **bold**. You won't (or shouldn't) see the `<b>` tags displayed, just the text rendered as bold. \n",
    "\n",
    "Many websites have tons of pages and content. These are likely not saved as separate individual files. The HTML to display those many pages is usually coming from some form of a template. The tool reads the data from the database, and sends it to the template, the template, then generates the HTML with the contents to be displayed. This is a really simplified way explaining it, and there are many many tools that do these sorts of things.\n",
    "\n",
    "The important thing to remember is that when you're dealing with , is that the data is being stored somewhere with some structure. You won't always know how or with what, but you can usually get a good idea about it by looking at the HTML being generated to display it. The HTML formatting elements will often have content specific tags about what it is displaying. For example if it's about a person, they may want the name displayed in a certain way. There are special ways to define HTML formatting where you can label specific groups of content, like a name. So when you look inside the HTML, you may see the name surrounded by some formatting tag that literally says \"name\". Not always, but generally, you can use these formatting labels to quite clearly extract the data that you want.\n",
    "\n",
    "# The core tools\n",
    "We'll be talking first about simpler techniques, as they often will enough to get you going. Then move into the larger tools, like using requests to download things, pathlib to manage the files, and then using regex and xpath to extract information.\n",
    "\n",
    "* `requests` a very popular and well supported library for handling http calls. Commonly used to read and download website content or files.\n",
    "* `pathlib` a modern object oriented way of handling files in python, managing folders, etc. Very much an essential tool even if it lacks some of the big buzz words.\n",
    "* `re` this is the Python regular expressions module. It is used to match text patterns within free text\n",
    "* `lxml` this package is used to help parse xml and html files and what we will use to execute some xpath queries\n",
    "* `bs4` this is the beautiful soup package, and it used for cleaning up messy html. It can be used to extract content if you want, but xpath queries are more powerful\n",
    "* `csv` and `json` these are two python packages we will use to export our data out\n",
    "\n",
    "# Simpler forms of web scraping\n",
    "In many cases, you may just want to copy an HTML table into something that is actionable data. This may be a CSV file, Excel file, or maybe some thing that you read directly into Python. There are a variety of tools to help you take a single HTML table and get it into one of those things.\n",
    "\n",
    "## Copy/paste it into a spreadsheet\n",
    "\n",
    "### A simple HTML table\n",
    "\n",
    "* http://www.neoperceptions.com/snakesandfrogs.com/scra/ident/names.htm\n",
    "\t* Open the page and look in the view source. \n",
    "\t* Looking at the table we can see this HTML comment\n",
    "```\n",
    "<!-- The following table was generated by the Internet Assistant Wizard for Microsoft Excel. -->\n",
    "<!-- ------------------------- -->\n",
    "<!-- START OF CONVERTED OUTPUT -->\n",
    "<!-- ------------------------- -->0\n",
    "```\n",
    "So this content is likely not being served up to us by another tool, but this is just plain HTML. Say that we want this back into a data file. \n",
    "\n",
    "We can select the text within the table, copy, and paste it into Excel or Sheets. However, note that the styling will also get pasted in. \n",
    "\n",
    "### More complex table\n",
    "\n",
    "Let's look at another one. https://threadcolors.com Looking at the scale of this and the HTML all being horizontal, we can safely say that something is generating this html. You can also see the javascript stuff in there as well. \n",
    "\n",
    "Let's copy/paste the first part of this table in and see what happens. On my computer, Excel doesn't paste the colors, but google sheets does. I've also seen pictures and other things get pasted in, along with font styling and other formatting. \n",
    "\n",
    "### Suppress styling with \"Paste special\"\n",
    "\n",
    "Excel and Sheets have versions of \"paste special\". There are some really nice extra tools in here if you've never explored. I'll briefly explain where to find these things, but interfaces and versions always change. \n",
    "\n",
    "* Excel has a few places for finding this. I like to right click on the top left cell where it should go, then select \"paste special\".\n",
    "\t* You'll see some options there, including another Paste Special. Choosing that opens up a window where you can choose HTML, Unicode text, or text. \n",
    "\t* Generally I choose \"text\" to get just the plain text. \n",
    "\n",
    "Microsoft and Google each have data import tools etc. you can also play with. \n",
    "\n",
    "## Google sheets importing tools\n",
    "\n",
    "Type `=IMPORT` into a google sheets cell and you'll see a bunch of options. These include tools for reading in data files online etc. Let's look at `importhtml`.\n",
    "\n",
    "https://support.google.com/docs/answer/3093339?hl=en\n",
    "\n",
    "Functions like these can be really handy, but you need to work with them really closely. These functions can work really nicely, but expect to spend some time playing with the arguments to ensure it's working correctly.\n",
    "\n",
    "This will take a URL along with other arguments and import the specified table of data into your sheet. \n",
    "\n",
    "The second argument is labeled query, but is asking you to specify if they should be importing a list of data or a table of data. We want to specify table.\n",
    "\n",
    "The index argument is asking you to specify which table on the website, it should import. Some pages may have dozens of tables, so youcount from the top down and provided the number (starting at 1) for which table it should be. Always check to ensure the right one has come in, because the way the tables appear on the website may not exactly match other specified in the HTML if there is this that many tables on the page that you may accidentally hit the wrong one.\n",
    "\n",
    "Here is our cell argument: `=IMPORTHTML(\"http://www.neoperceptions.com/snakesandfrogs.com/scra/ident/names.htm\",\"table\",1)`\n",
    "\n",
    "Take a moment to look at how the data has been read into your sheet. The upper left cell, where you originally put the function information, will still contain the function content, but the other cells will only have the actual text contact. You may also noticed that some of the formatting, like bold and italics did not come through correctly. \n",
    "\n",
    "Tools like this can extend how powerful this idea copy and pasting into a spreadsheet can be. Say you had a single table with basic formatting, and wanted to import it into a spreadsheet regularly. Using a function like this would allow you to incorporate some amount of automation, where this can be repeated without having to copy paste the table each time. However, there are several limitations. You shouldn't use this without observing the results in case the original website structure has changed. Timestamp access information is also not retained. \n",
    "\n",
    "The list option is also quite interesting. However, it is only importing the top most list that it sees. Meaning that navigating a deeper structure can be harder. \n",
    "`=IMPORTHTML(\"https://en.wikipedia.org/wiki/Lists_of_American_universities_and_colleges\", \"list\", 6)`\n",
    "\n",
    "TIPS: you'll also see `importxml` in this list, allowing you to do easy xpath queries. This won't give you all the power of using xpath with python, but is still extremely useful. Xpath will be covered more later on. \n",
    "\n",
    "`=IMPORTXML(\"https://en.wikipedia.org/wiki/Lists_of_American_universities_and_colleges\", \"//div[@class = 'mw-parser-output']/ul/li\")` \n",
    "\n",
    "Looking at the results, we can still see some limitations.\n",
    "\n",
    "`=IMPORTXML(\"https://en.wikipedia.org/wiki/List_of_colleges_and_universities_in_Illinois\", \"//table[contains(@class, 'wikitable')]/tbody/tr/th/a\")`\n",
    "\n",
    "If you play around more you can get more, but generally this won't be quite enough.\n",
    "\n",
    "## Summary\n",
    "\n",
    "In the section, we saw a few tools to copy and paste or import contacts in from webpages or tables. These tools tend to be really useful for smaller tasks or exploration, but can't (okay shouldn't) be taken much further than that.\n",
    "\n",
    "# Downloading things\n",
    "\n",
    "Now we're getting into a situation where we have a list of things we need to download.\n",
    "\n",
    "This exact situation will depend on what you're doing. I've done this where each page had 50 PDF links that I needed to download and there were 20 total pages. You may also have a page with 100 images and you want to programmatically download all of the images on that page. There's so many reasons that you might want to do this, but the nice thing is this sort of task is a great initial web scraping task.\n",
    "\n",
    "This sort of task also lens itself really nicely to combining both manual work and programmatic work. This situation may dictate which one gets which, but is there a lot of flexibility.\n",
    "\n",
    "Your first step here is going to identify where is the page that has all of the links on it and then you need to get access to all of them. My general preference is to download and save this page to my computer, because that allows me to experiment with par seeing and figure things out on my own time without having to reload the page or hit their server for every time that I run my script.\n",
    "\n",
    "Second, you need a proof of concept to check that you can actually get the content out from those pages. Use just one page to experiment.\n",
    "\n",
    "Tip: you can often be working on these experiments with getting content out while the other pages you need are downloading.\n",
    "\n",
    "## Before we move on, some considerations\n",
    "\n",
    "There are a few things to consider before we move on where we are programmatically downloading things from someone else's server. Not every website wants to be scraped. Some have restrictions some have blocks, and there's a certain kind of etiquette that we want to follow.\n",
    "\n",
    "First, we want to keep the speed we are hitting their server to something reasonable. This is usually a minimum of 4 seconds, but I've worked with pages that asked for 30 seconds delays. \n",
    "\n",
    "Second, some ask that you only do large scale harvesting or scraping during \"off peak\" times. This often means overnight.\n",
    "\n",
    "Third, some pages may just completely ban scraping tools from being used. Usually this is because they have an API they'd prefer you to use (and usually pay for) or because the data is sensitive in some way.  Let's look a few examples. \n",
    "\n",
    "* Linkedin has a hard block on programmatic web scraping because their data is really valuable and they want to sell it to you.\n",
    "* Google will quickly block you from scraping their results because they want to you to use an API. Many of theirs are open and reasonable to use, but they don't want HTML scraping.\n",
    "* Archive of Our Own (AO3) has a block against it because they don't want search engines to index the results. This gives them control over story and author information and the ability to fully take things down as needed. \n",
    "\n",
    "But how can you know for sure? This can be hard and there's no single answer. You can often check the `robots.txt` file for the website. You can read about this file here: https://en.wikipedia.org/wiki/Robots.txt Very generally, it will contain information for humans and for bots, and give you an idea about limitations, etc. Not every site will have it, but most with data will. \n",
    "\n",
    "* https://en.wikipedia.org/robots.txt\n",
    "* https://archiveofourown.org/robots.txt\n",
    "\t* my favorite \"cruel but efficient\"\n",
    "\t* note the crawl delay\n",
    "* https://www.fanfiction.net/robots.txt\n",
    "\n",
    "You can ask for permission to go out of bounds for this, especially for research. Just be respectful.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca63e4f9-5fff-47e5-af90-50131c877033",
   "metadata": {},
   "source": [
    "## Handling delays\n",
    "\n",
    "Most programming languages will have some ability to \"delay\"actions. We will use the `time` module in Python to delay our execution.\n",
    "\n",
    "`time.sleep(seconds)` takes a number of seconds and pauses script execution for that long. Other languages use `ms` instead, so be mindful if switching!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "4521e05f-9f51-490f-814b-7f2b9e091ad7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hello!\n",
      "hello!\n",
      "hello!\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "for _ in range(3):\n",
    "\tprint(\"hello!\")\n",
    "\ttime.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c2ac2ca-f818-4129-b2ed-e0c911f17626",
   "metadata": {},
   "source": [
    "## Downloading things off one page\n",
    "\n",
    "Starting with the simplest version for sure, we have one page with a side of links, and we want to download the results of those links. What those files are, doesn't really matter because you're downloading them to disk. \n",
    "\n",
    "So what I love about this page is that they just have the sql statement right at the top of the page. \n",
    "\n",
    "https://calphotos.berkeley.edu/cgi/img_query?where-taxon=Allium+anceps\n",
    "\n",
    "Let's take a look at the structure here:\n",
    "\n",
    "* clearly these are coming from a database\n",
    "* there are multiple pages\n",
    "* the images are displayed on the page\n",
    "* there are detail links by each image\n",
    "* being displayed in a table\n",
    "\n",
    " Tip: Chrome XPath Helper tool\n",
    "\n",
    "I like to use this to preview the structure of the elements.\n",
    "\n",
    "There are a variety of tools you can use for this part! Our basic goal for this is to get URL for each of the pictures. Once we have those collected, we can run through them to download each. I'm going to provide these URLs for now so we can focus on the downloading. \n",
    "\n",
    "Just a small preview of this xpath we'll be using:\n",
    "\n",
    "`//td//img/@src`\n",
    "\n",
    "* we can use `//img` to get all the images on the page, but most pages will have other images. Best practice is to include something more specific to disambiguate. This is why I have `td` in here.\n",
    "* Using `@src` allows me to request that it return the value for the source property\n",
    "* the URLs for the images appear to have a specific folder structure, which I could have also used to gather them\n",
    "* the URLs gathered are relative links, meaning that I'll need to build the full URL when I'm doing my pass over them. \n",
    "\n",
    "Let's open the text file with the URLs and start building those up. As mentioned, these are relative links so we will need to do a bit of editing to get them into the full pattern.  You can check out a link on the main page to inspect what the full URL should be and what the relative links are. Looking a that we can discover that the \"base\" url should be.\n",
    "\n",
    "Here's a full link:\n",
    "`https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1209/2448.jpeg`\n",
    "\n",
    "And here's the corresponding relative link: \n",
    "\n",
    "`imgs/128x192/0000_0000/1209/2448.jpeg`\n",
    "\n",
    "This means we'll need to prepend `https://calphotos.berkeley.edu` before each URL to have the full one. There are several ways you can do this and this is a great time to practice your core Python skills. \n",
    "\n",
    "Some notes:\n",
    "\n",
    "* using list comprehension syntax here\n",
    "* using `readlines` to read it in, which returns a list of strings, each string is a line from the file plus a newline character\n",
    "* `strip` is needed to take the ending newline character off\n",
    "* I'm concatenating the base before the url from the line, but note that I didn't include the final / because there's already an opening one from the url. \n",
    "* This will result in a list of all the urls."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "62ad925f-1e67-414d-9376-86214eaf02d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('pictures.txt', 'r', encoding = 'utf-8') as infile:\n",
    "    # urls = infile.readlines()\n",
    "    urls = ['https://calphotos.berkeley.edu' + u.strip() for u in infile.readlines()]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "7f4eac19-9e37-422c-b5e3-b112f11da7a1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0903/0732.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1002/0400.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1102/0790.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1102/0792.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1207/0067.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1207/0083.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1207/0084.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1207/0086.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0408/1095.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2437.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2438.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2439.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2440.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2441.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2442.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2443.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0608/2444.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0209/0663.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0209/0664.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0209/0665.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0209/0666.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0209/0667.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0509/0139.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1209/2447.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1209/2448.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0611/1218.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0611/1219.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0611/1220.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0611/1221.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0611/1222.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0611/1223.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0413/3699.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1113/3030.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1115/2820.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1115/2821.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1115/3063.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1115/3064.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1017/1587.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1017/1588.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/1017/1589.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0918/2740.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0918/2741.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0918/2742.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0918/2743.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0918/2744.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0720/2473.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0720/2474.jpeg',\n",
       " 'https://calphotos.berkeley.edu/imgs/128x192/0000_0000/0720/2475.jpeg']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urls"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab181484-670c-42a2-a51a-732faabcb4fb",
   "metadata": {},
   "source": [
    "### Working with `requests`\n",
    "\n",
    "Let's try something basic!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4d6aaeb0-61ca-4c7f-a63e-040fe46ec183",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Response [200]>\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "url = \"https://loripsum.net/api/1/plaintext/short\"\n",
    "result = requests.get(url)\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6581588-9162-4d31-ba5d-aeb5eca6c4d4",
   "metadata": {},
   "source": [
    "\n",
    "So what we're seeing here is a sucessfull connection, but not the text.  We have to ask about that explicitly from out result object.\n",
    "\n",
    "We do this with `.text` (no parens!) this will allow us to ask for a variable value within out object (versus calling a function). Some objects just work this way, and we know how to do this by looking at the documentation or a tutorial.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1aac997-89f9-4259-bcdb-5c04469188a6",
   "metadata": {},
   "source": [
    "But our items are images? What can we do. Just a few tweaks. From python's perspective, we are moving from data that's text to data that's bytes. \n",
    "\n",
    "`requests` actually has a bunch of ways to handle this, and those methods may be better for larger files, etc. However, for smaller files like ours and the fact that we are using pathlib... we can pretty easily handle this. \n",
    "\n",
    "### Working with `pathlib`\n",
    "\n",
    "You'll note that I didn't use pathlib for reading in that file. That's okay! Sometimes you don't need to.\n",
    "\n",
    "`pathlib` is a great module for working with files/folders/etc. For webscraping it is ideal because you can very cleanly handle making folders, checking if things exist, making longer file paths, etc. Honestly, when I started using it vs other tools it was game changing.\n",
    "\n",
    "We'll be exploring things with `pathlib` as we go, but we do need to cover a few basics. \n",
    "\n",
    "You create `Path` objects to represent files and directories. Once these are made you get access to special methods for taking action on them or getting information back. You create a file and a folder object the same way.\n",
    "\n",
    "`p = pathlib.Path(string of path info etc.)` \n",
    "\n",
    "This returns a `Path` object you'll want to save as an object. \n",
    "\n",
    "We can use `pathlib.Path('pictures')` to work with our directory and then make the file path objects like `pathlib.Path('filename.jpeg')`. Neither of these things need to actually exist for us to make these objects. \n",
    "\n",
    "We can use the `mkdir()` method to create a folder, and then use the `/` concatenation operator to combine them.\n",
    "\n",
    "`pathlib` has two awesome path object methods to write out content:\n",
    "\n",
    "* `write_text(text stuff)``\n",
    "\t* for text!\n",
    "* `write_bytes(a bytes or non-text doodad)`\n",
    "\t* briefly, for stuff that isn't text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0340553-bf50-4268-ab30-7315cc3c4d1a",
   "metadata": {},
   "source": [
    "\n",
    "https://calphotos.berkeley.edu/robots.txt\n",
    "\n",
    "We have a list of URLs now, so we can loop through those and begin downloading them. There are a few tasks we'll need to accomplish.\n",
    "\n",
    "* create the file name (from the file name)\n",
    "* create a directory for the new files to go into\n",
    "* create the full destination path (target folder plus file name)\n",
    "* open up the requests connection\n",
    "* access and write the content\n",
    "* close the connection\n",
    "* wait for 5 seconds\n",
    "\n",
    "This is a lot and we build it up bit by bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "87b48ff7-4540-4596-84a9-3e046428b09d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pictures/0903_0732.jpeg\n",
      "pictures/1002_0400.jpeg\n",
      "pictures/1102_0790.jpeg\n",
      "pictures/1102_0792.jpeg\n",
      "pictures/1207_0067.jpeg\n",
      "pictures/1207_0083.jpeg\n",
      "pictures/1207_0084.jpeg\n",
      "pictures/1207_0086.jpeg\n",
      "pictures/0408_1095.jpeg\n",
      "pictures/0608_2437.jpeg\n",
      "pictures/0608_2438.jpeg\n",
      "pictures/0608_2439.jpeg\n",
      "pictures/0608_2440.jpeg\n",
      "pictures/0608_2441.jpeg\n",
      "pictures/0608_2442.jpeg\n",
      "pictures/0608_2443.jpeg\n",
      "pictures/0608_2444.jpeg\n",
      "pictures/0209_0663.jpeg\n",
      "pictures/0209_0664.jpeg\n",
      "pictures/0209_0665.jpeg\n",
      "pictures/0209_0666.jpeg\n",
      "pictures/0209_0667.jpeg\n",
      "pictures/0509_0139.jpeg\n",
      "pictures/1209_2447.jpeg\n",
      "pictures/1209_2448.jpeg\n",
      "pictures/0611_1218.jpeg\n",
      "pictures/0611_1219.jpeg\n",
      "pictures/0611_1220.jpeg\n",
      "pictures/0611_1221.jpeg\n",
      "pictures/0611_1222.jpeg\n",
      "pictures/0611_1223.jpeg\n",
      "pictures/0413_3699.jpeg\n",
      "pictures/1113_3030.jpeg\n",
      "pictures/1115_2820.jpeg\n",
      "pictures/1115_2821.jpeg\n",
      "pictures/1115_3063.jpeg\n",
      "pictures/1115_3064.jpeg\n",
      "pictures/1017_1587.jpeg\n",
      "pictures/1017_1588.jpeg\n",
      "pictures/1017_1589.jpeg\n",
      "pictures/0918_2740.jpeg\n",
      "pictures/0918_2741.jpeg\n",
      "pictures/0918_2742.jpeg\n",
      "pictures/0918_2743.jpeg\n",
      "pictures/0918_2744.jpeg\n",
      "pictures/0720_2473.jpeg\n",
      "pictures/0720_2474.jpeg\n",
      "pictures/0720_2475.jpeg\n"
     ]
    }
   ],
   "source": [
    "import pathlib\n",
    "import time\n",
    "import requests\n",
    "\n",
    "# create the target folder object\n",
    "target = pathlib.Path('pictures')\n",
    "# make the directory if needed\n",
    "# does nothing if already exists\n",
    "target.mkdir(exist_ok=True)\n",
    "\n",
    "for u in urls:\n",
    "    parts = u.split('/')\n",
    "    last_two = parts[-2:] # grab the last two parts\n",
    "    fname = \"_\".join(last_two)\n",
    "    # print(fname)\n",
    "    p = target / pathlib.Path(fname)\n",
    "    print(p) # this is the full path\n",
    "    r = requests.get(u) #open connection\n",
    "    p.write_bytes(r.content) # get content, write bytes\n",
    "    r.close() # always close your connection!!!\n",
    "    time.sleep(5) # pause to not anger the server"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0474aa7d-faff-4874-b07b-eb9ab6ea221b",
   "metadata": {},
   "source": [
    "One thing I always check at this point is the file size for everything that has downloaded. When in jupyter on a cloud service, that can be hard, but `!` to the rescue.\n",
    "\n",
    "`!ls -l pictures`\n",
    "\n",
    "Now, what if we had many or some messed up? Using pathlib is awesome here. We can utilize the `exists()` method to check if the file we are proposing to make already exists. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "3863a3fc-b2f3-4597-b76c-9c5931ed9317",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total 5384\n",
      "-rw-r--r--  1 wickes1  staff  79560 Jul 24 11:27 0209_0663.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  77499 Jul 24 11:27 0209_0664.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  70946 Jul 24 11:27 0209_0665.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  61068 Jul 24 11:27 0209_0666.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  72332 Jul 24 11:27 0209_0667.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  48150 Jul 24 11:26 0408_1095.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  41317 Jul 24 11:28 0413_3699.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  53845 Jul 24 11:27 0509_0139.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  92166 Jul 24 11:26 0608_2437.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  81895 Jul 24 11:26 0608_2438.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  79842 Jul 24 11:26 0608_2439.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  74976 Jul 24 11:26 0608_2440.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  71964 Jul 24 11:26 0608_2441.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  65093 Jul 24 11:26 0608_2442.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  61289 Jul 24 11:26 0608_2443.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  82562 Jul 24 11:27 0608_2444.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  30007 Jul 24 11:27 0611_1218.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  24398 Jul 24 11:27 0611_1219.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  29471 Jul 24 11:28 0611_1220.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  29648 Jul 24 11:28 0611_1221.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  28314 Jul 24 11:28 0611_1222.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  26117 Jul 24 11:28 0611_1223.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  45154 Jul 24 11:29 0720_2473.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  47276 Jul 24 11:29 0720_2474.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  42571 Jul 24 11:29 0720_2475.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  50755 Jul 24 11:25 0903_0732.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  62078 Jul 24 11:29 0918_2740.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  56500 Jul 24 11:29 0918_2741.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  57571 Jul 24 11:29 0918_2742.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  56558 Jul 24 11:29 0918_2743.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  54161 Jul 24 11:29 0918_2744.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  53597 Jul 24 11:25 1002_0400.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  40324 Jul 24 11:28 1017_1587.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  44893 Jul 24 11:28 1017_1588.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  44588 Jul 24 11:29 1017_1589.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  48369 Jul 24 11:25 1102_0790.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  41303 Jul 24 11:25 1102_0792.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  43702 Jul 24 11:28 1113_3030.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  49867 Jul 24 11:28 1115_2820.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  42713 Jul 24 11:28 1115_2821.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  49717 Jul 24 11:28 1115_3063.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  43975 Jul 24 11:28 1115_3064.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  72338 Jul 24 11:25 1207_0067.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  87309 Jul 24 11:26 1207_0083.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  89846 Jul 24 11:26 1207_0084.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  79006 Jul 24 11:26 1207_0086.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  38645 Jul 24 11:27 1209_2447.jpeg\n",
      "-rw-r--r--  1 wickes1  staff  32848 Jul 24 11:27 1209_2448.jpeg\n"
     ]
    }
   ],
   "source": [
    "!ls -l pictures"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c47c4ec4-106e-49a8-b9df-22102c301634",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n",
      "already done!\n"
     ]
    }
   ],
   "source": [
    "target = pathlib.Path('pictures')\n",
    "target.mkdir(exist_ok=True)\n",
    "\n",
    "for u in urls:\n",
    "    parts = u.split('/')\n",
    "    last_two = parts[-2:] # grab the last two parts\n",
    "    fname = \"_\".join(last_two)\n",
    "    p = target / pathlib.Path(fname)\n",
    "    # use .exists to check\n",
    "    if p.exists():\n",
    "        print(\"already done!\")\n",
    "    else:\n",
    "        print(p)\n",
    "        r = requests.get(u)\n",
    "        p.write_bytes(r.content)\n",
    "        r.close()\n",
    "        time.sleep(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eca42822",
   "metadata": {},
   "source": [
    "# Exercises (Optional)\n",
    "\n",
    "`If possible, include practice exercises for users to do on their own. These may have clear solutions or be more open-ended.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11131305-7648-49f2-93be-bd3326bc4694",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e42c9736",
   "metadata": {},
   "source": [
    "# Solutions (Optional)\n",
    "`Offer some possible solutions for the practice exercises.`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "227a6d92",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d2368107",
   "metadata": {},
   "source": [
    "# References (Optional)\n",
    "No citations required but include this if you have cited academic sources. Use whatever format you like, just be consistent. Markdown footnotes are not well-supported in notebooks.[$^{1}$](#1) I suggest using an anchor link with plain html as shown.[$^{2}$](#2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c7fd8c",
   "metadata": {},
   "source": [
    "1. <a id=\"1\"></a> Here is an anchor link footnote.\n",
    "2. <a id=\"2\"></a> D'Ignazio, Catherine and Lauren F. Klein. [*Data Feminism*](https://mitpress.mit.edu/books/data-feminism). MIT Press, 2020."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}