Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limiting concurrency #282

Closed
callumlocke opened this issue Oct 11, 2016 · 3 comments
Closed

Limiting concurrency #282

callumlocke opened this issue Oct 11, 2016 · 3 comments

Comments

@callumlocke
Copy link

I'm trying to take 1000 screenshots of different URLs on my localhost, and running into problems.

// option 1 - multiple instances
for (const url of urls) {
  await new Pageres().src(url).dest(__dirname).run();
}

// option 2 - single instance
const pageres = new Pageres();
for (const url of urls) pageres.src(url);
await pageres.dest(__dirname).run();

The problem with option 1 is it takes forever (I guess because it spins up a new Phantom.js for every URL).

The problem with option 2 is it seems to hit my local web server with 1000 requests all at the same time, which it can't handle.

I want to do the option 2 approach, but throttled so it only does 10 requests at any one time. Something like const pageres = new Pageres({ concurrency: 10 });. Would that be possible?

@SamVerschueren
Copy link
Contributor

Duplicate of #242.

One of the suggested solutions is to use promise-concurrent or some other library that can handle that.

@callumlocke
Copy link
Author

But that's not a good solution because it involves making multiple PhantomJS processes, each of which has a significant startup time and memory overhead.

By sharing a single PhantomJS process you can save a huge amount of memory and time for large batches of URLs.

I am willing to do a PR for this if you would consider it.

@kevva
Copy link
Contributor

kevva commented Oct 11, 2016

@callumlocke, you'd have to do it in screenshot-stream I think. I'm not sure how you'd change the URL while keeping the process alive.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants