Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate Zyte related code from core #382

Open
4 tasks
rennerocha opened this issue Jan 24, 2023 · 0 comments · May be fixed by #383
Open
4 tasks

Separate Zyte related code from core #382

rennerocha opened this issue Jan 24, 2023 · 0 comments · May be fixed by #383
Milestone

Comments

@rennerocha
Copy link
Collaborator

Considering that Spidermon was created at Zyte (former Scrapinghub), it is understandable that we have monitors and actions that are related to services provided by it. However, considering that Spidermon can be used by users that are not running their spiders at Zyte infrastructure, it would be a good refactoring to keep every code that is Zyte-related only inside a separated module.

My first suggestion is to move everything (actions, monitors, mixins and utils) that are spread in the project into a separated spidermon.contrib.zyte.* module, so it will be clear to everyone that anything imported from this module is related to Zyte and may require that the extension is being executed inside Zyte infrastructure.

I found Zyte-only related code in the following files:

  • spidermon/spidermon/utils/zyte.py
  • spidermon/contrib/stats/statscollectors/sc_collections.py
  • spidermon/contrib/scrapy/monitors.py
  • spidermon/contrib/scrapy/monitors/mixins/job.py
  • spidermon/contrib/actions/jobs/tags.py

Tasks to accomplish

  • Create spidermon.contrib.zyte module
  • Move all related code to this module
  • Avoid backward incompatibility adding a deprecated warning if we try to import in old locations (but allows it)
  • Update documentation
@rennerocha rennerocha linked a pull request Jan 24, 2023 that will close this issue
@VMRuiz VMRuiz added this to the 1.20.0 milestone Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants