Limiting concurrency

[callumlocke]

I'm trying to take 1000 screenshots of different URLs on my localhost, and running into problems.

// option 1 - multiple instances
for (const url of urls) {
  await new Pageres().src(url).dest(__dirname).run();
}

// option 2 - single instance
const pageres = new Pageres();
for (const url of urls) pageres.src(url);
await pageres.dest(__dirname).run();

The problem with option 1 is it takes forever (I guess because it spins up a new Phantom.js for every URL).

The problem with option 2 is it seems to hit my local web server with 1000 requests all at the same time, which it can't handle.

I want to do the option 2 approach, but throttled so it only does 10 requests at any one time. Something like const pageres = new Pageres({ concurrency: 10 });. Would that be possible?

[SamVerschueren]

Duplicate of #242.

One of the suggested solutions is to use promise-concurrent or some other library that can handle that.

[callumlocke]

But that's not a good solution because it involves making multiple PhantomJS processes, each of which has a significant startup time and memory overhead.

By sharing a single PhantomJS process you can save a huge amount of memory and time for large batches of URLs.

I am willing to do a PR for this if you would consider it.

[kevva]

@callumlocke, you'd have to do it in screenshot-stream I think. I'm not sure how you'd change the URL while keeping the process alive.