Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Connecticut scraper #3

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Add Connecticut scraper #3

wants to merge 7 commits into from

Commits on Feb 22, 2014

  1. Add Q to package dependencies

    sukima committed Feb 22, 2014
    Configuration menu
    Copy the full SHA
    216e384 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2014

  1. Add CT scraper

    This is an initial stab at a scraper for Connecticut. The website is a
    bit awkward if not insane.
    
    This currently only handles the first 20 as it need subsequent requests
    to get more.
    sukima committed Feb 24, 2014
    Configuration menu
    Copy the full SHA
    70b6397 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3a9f48a View commit details
    Browse the repository at this point in the history
  3. Modularize and cleanup promise chain

    Sorry this commit blows. A little too much spike work going on and
    didn't clean up the patch well.
    
    Basically move the chain logic into functions. Saves results to an array
    higher in the scope. Pulls the total logic out of the processHTML function
    sukima committed Feb 24, 2014
    Configuration menu
    Copy the full SHA
    7f77bac View commit details
    Browse the repository at this point in the history
  4. Add ability to request subsequent pages

    Will use the total to batch up page requests and wait for them to finish.
    sukima committed Feb 24, 2014
    Configuration menu
    Copy the full SHA
    108b902 View commit details
    Browse the repository at this point in the history
  5. Disable the async requests

    Seems that the site uses some wacked session management that I haven't
    been able to crack yet. Disable the feature till it can be understood.
    
    This is a problem with proper crafting of the form-data not the scraper
    logic.
    sukima committed Feb 24, 2014
    Configuration menu
    Copy the full SHA
    22ff170 View commit details
    Browse the repository at this point in the history
  6. Remove commented code 💀

    sukima committed Feb 24, 2014
    Configuration menu
    Copy the full SHA
    8a1dca1 View commit details
    Browse the repository at this point in the history