Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feat.] build ChekPoint system for the scrapper. #9

Open
rajatkb opened this issue Feb 23, 2020 · 3 comments
Open

[Feat.] build ChekPoint system for the scrapper. #9

rajatkb opened this issue Feb 23, 2020 · 3 comments
Labels
easy GSSOC label for beginner tag enhancement New feature or request good first issue Good for newcomers gssoc20 GSSOC label for gscco20 tag help wanted Extra attention is needed

Comments

@rajatkb
Copy link
Owner

rajatkb commented Feb 23, 2020

In case of a sudden failure and crash, we don't want the process to restart and start scrapping from the beggining.

** REQUIREMENT **
1 . Add method of Database class inside db.py to store check point information
2. Let user implementing plugin decide how to use the checkpointing system.

Note: Database class uses pymongo and contains no data that is modified after creation of object. So it's safe to say it is thread safe.
@rajatkb rajatkb added easy GSSOC label for beginner tag good first issue Good for newcomers gssoc20 GSSOC label for gscco20 tag labels Feb 23, 2020
@agnes-sharan
Copy link

Hi, I'd like to work on this issue!

@rajatkb rajatkb changed the title Build ChekPoint system for the scrapper. [Feat] build ChekPoint system for the scrapper. Feb 28, 2020
@rajatkb rajatkb changed the title [Feat] build ChekPoint system for the scrapper. [Feat.] build ChekPoint system for the scrapper. Feb 28, 2020
@rajatkb rajatkb mentioned this issue Mar 4, 2020
Open
@rajatkb
Copy link
Owner Author

rajatkb commented Mar 9, 2020

@agnes-sharan I have not received any updated about this issue. If you would not be attending to it , please inform me. I will have it reassigned.

@rajatkb rajatkb assigned rajatkb and unassigned agnes-sharan and rajatkb Mar 12, 2020
@rajatkb
Copy link
Owner Author

rajatkb commented Mar 12, 2020

@agnes-sharan , removing from asignee for the time being. Incase anybody else can attend to it. If later you can pick it, will reassign the issue 💯

@rajatkb rajatkb added enhancement New feature or request help wanted Extra attention is needed labels Mar 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
easy GSSOC label for beginner tag enhancement New feature or request good first issue Good for newcomers gssoc20 GSSOC label for gscco20 tag help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@rajatkb @agnes-sharan