Scraping

[irthomasthomas]

No description provided.

[irthomasthomas]

https://github.com/platisd/sycophant/blob/master/sycophant.py
Good example of scraping > summarization > publishing pipeline.

[irthomasthomas]

If you weren't already aware, Scrapy has strong support for this via their HTTPCache middleware; you can choose whether to have it actually behave like a cache, choosing to returned already scraped content if matched or merely to act as a pass-through cache: https://docs.scrapy.org/en/2.7/topics/downloader-middleware....

Their OOtB storage does what the sibling comment says about sha1-ing the request and then sharding the output filename by the first two characters: https://github.com/scrapy/scrapy/blob/2.7.1/scrapy/extension...

https://news.ycombinator.com/item?id=33564138

[irthomasthomas]

• https://github.com/irthomasthomas/help-scraper/blob/3b7dc2070f7c995a394e6d044a52d2605e7b8e2a/.github/workflows/scrape.yml

https://github.com/irthomasthomas/help-scraper/blob/3b7dc2070f7c995a394e6d044a52d2605e7b8e2a/.github/workflows/scrape.yml#L1-L109

[irthomasthomas]

• Possible to to try to extract main article from a page? alirezamika/autoscraper#86
• How to scrape a dynamic website? alirezamika/autoscraper#71

[irthomasthomas]

List of crawling and scraping tools. Not updated for 6 years, though.

• [] https://github.com/darpa-i2o/memex-program-index

[irthomasthomas]

curl -s https://api.github.com/repos/jiaaro/pydub/commits | jq -r '.[0] |  .commit.message, .commit.author.date'