Add Connecticut scraper #3
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is an initial stab at a scraper for Connecticut. The website is a bit awkward if not insane. This currently only handles the first 20 as it need subsequent requests to get more.
Sorry this commit blows. A little too much spike work going on and didn't clean up the patch well. Basically move the chain logic into functions. Saves results to an array higher in the scope. Pulls the total logic out of the processHTML function
Will use the total to batch up page requests and wait for them to finish.
Seems that the site uses some wacked session management that I haven't been able to crack yet. Disable the feature till it can be understood. This is a problem with proper crafting of the form-data not the scraper logic.
|
Awesome!! 🌠 You want me to just leave this PR open for you to add more commits? |
|
Sure unless you know how to fix it. Sigh I gave up last night. |
|
K -- I'll take a look but it could take me a while, since I'm back in day-job mode here at DOBT. That postdata looks absolutely ridiculous. The other option is to use a headless browser instead of crafting the requests manually -- maybe that's worth a shot if we can't get this method to paginate. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a bit of WIP. It will scrape the first 20. I can't seem to craft the following page requests' form-data correctly and each page after the first are returning the wrong page.
P.S. The site / markup is scary as all hell 😱