Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve speed of DynamicFetcher.Fetch for pagination #253

Open
jakopako opened this issue Nov 18, 2023 · 0 comments
Open

Improve speed of DynamicFetcher.Fetch for pagination #253

jakopako opened this issue Nov 18, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@jakopako
Copy link
Owner

Currently, with dynamic websites, when page number N is fetched the scraper fetches the initial url and then "clicks" on the next-page button N times. So page 0 means fetch and click 0 times, page 1 fetch and click 1 time, page 2 fetch and click 2 times. Since there is a short delay between each click, fetching many pages can take quite some time.

This can probably be improved if we store the intermediate state somehow. Then for each page we'd only need to "click" once on the next-page button.

@jakopako jakopako added the enhancement New feature or request label Nov 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant