This repository has been archived by the owner on Mar 17, 2020. It is now read-only.
API scraper #3
Open
API scraper #3
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…s. it's highly concurrent, but there is a lot of room for optimisation improvements. Signed-off-by: Mike Lloyd <[email protected]>
Signed-off-by: Mike Lloyd <[email protected]>
Signed-off-by: Mike Lloyd <[email protected]>
Signed-off-by: Mike Lloyd <[email protected]>
added nil pointer checks due to dereference panics. added version regular expressions for later. added global counter for humanity's sake. split out version table buffer from code block buffer. removed remarks regular expression due to garbage data. there's still an issue with deadlocks/race condition somewhere, but it's really inconsistent. sometimes it stalls at like ~3k functions found, other times it stalls at like ~17k functions found. I don't know where the problem is. I'm wondering if it's on the Microsoft side, due to all of the crawling traffic. I wonder if it might be a DoS concern. Signed-off-by: Mike Lloyd <[email protected]>
|
Welp, I got blacklisted from the docs website, haha.
|
|
I need to add a rate limiter. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
adding the docs scraper to go through and express all the Windows APIs. it's highly concurrent, but there is a lot of room for optimisation improvements. currently I've only tested it for Windows Desktop, but once I flesh out the last big of bugs, it should be good to go.
@aaronmsft @bitcrazed @erikstmartin this should really help grease the wheels on some of our conversations.
Signed-off-by: Mike Lloyd [email protected]