Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CityProtect scraper rework #231

Open
EvilDrPurple opened this issue Sep 19, 2023 · 0 comments
Open

CityProtect scraper rework #231

EvilDrPurple opened this issue Sep 19, 2023 · 0 comments

Comments

@EvilDrPurple
Copy link
Contributor

Context

The CityProtect scraper could use a rework. It works well enough in what it does but is outdated and the instructions in the README are confusing

Requirements

  • Should work the same as it currently does
  • Should use functional programming so it can be called upon by scrapers as a module
  • The configs.py method should be scrapped in favor of passing function parameters to make things more straightforward
  • There should be a template detailing how a user may call upon the module in their own scraper. See Open Data Template
  • The CityProtect scraper and CityProtect_Bulk script should probably live in one folder together
  • Reworked README
  • Cleanup the old or useless comments

Docs

  • The README should be clear and concise in how to setup and run the CityProtect scraper

Open questions

  • There's a lot of adding and condition statements to retrieve file names are urls. Perhaps this could be reworked with datetime somehow to make it simpler?
  • The CityProtect_Bulk scraper seems to use an API, perhaps this could be looked into implementing into the CityProtect scraper as well?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant