-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Connecticut scraper #3
Open
sukima
wants to merge
7
commits into
dobtco:master
Choose a base branch
from
sukima:feature/ct
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Commits on Feb 22, 2014
-
Configuration menu - View commit details
-
Copy full SHA for 216e384 - Browse repository at this point
Copy the full SHA 216e384View commit details
Commits on Feb 24, 2014
-
This is an initial stab at a scraper for Connecticut. The website is a bit awkward if not insane. This currently only handles the first 20 as it need subsequent requests to get more.
Configuration menu - View commit details
-
Copy full SHA for 70b6397 - Browse repository at this point
Copy the full SHA 70b6397View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3a9f48a - Browse repository at this point
Copy the full SHA 3a9f48aView commit details -
Modularize and cleanup promise chain
Sorry this commit blows. A little too much spike work going on and didn't clean up the patch well. Basically move the chain logic into functions. Saves results to an array higher in the scope. Pulls the total logic out of the processHTML function
Configuration menu - View commit details
-
Copy full SHA for 7f77bac - Browse repository at this point
Copy the full SHA 7f77bacView commit details -
Add ability to request subsequent pages
Will use the total to batch up page requests and wait for them to finish.
Configuration menu - View commit details
-
Copy full SHA for 108b902 - Browse repository at this point
Copy the full SHA 108b902View commit details -
Seems that the site uses some wacked session management that I haven't been able to crack yet. Disable the feature till it can be understood. This is a problem with proper crafting of the form-data not the scraper logic.
Configuration menu - View commit details
-
Copy full SHA for 22ff170 - Browse repository at this point
Copy the full SHA 22ff170View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8a1dca1 - Browse repository at this point
Copy the full SHA 8a1dca1View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.