[performance] Compress in parallel

[jaymakes11]

Love this package!

But not the performance, especially when using it with lots of image files :(

It doesn't seem to be compressing in parallel.

A "make parallel" improvement to Astro's build command was made here. Perhaps something similar could be done here to improve the performance.

From a quick look at the code, perhaps the current bottleneck somewhere here?

Happy to take a stab at making some improvements on this once I can free-up some time.

[jaymakes11]

For context, I'm seeing astro-compress take around 30 minutes to compress ~1,000 images (on an M1 Max).

1,000 images might seem like a lot, but current responsive/retina best practices result in lots of image variants (the site in reference here only has 36 source images).

[NikolaRHristov]

Thank you, I've been wondering about how to make it compress in parallel!

[NikolaRHristov]

Expect it in 1.2.0

[NikolaRHristov]

So I did a benchmark with three different methods https://github.com/nikolaxhristov/bench-piscina I think piscina might be the best for this.

[johnzanussi]

@nikolaxhristov I would love to help out with this if it's something you want help with.

[NikolaRHristov]

@johnzanussi TY! ❤️ I would love some help. I wanted to make the files-pipe component itself parallelized, so whatever you use in callbacks gets called parallelly. This is used by astro-compress, astro-rome, astro-critters and a few other extensions, so it would greatly benefit the whole ecosystem.

Ideally, this would be parallelized https://github.com/lightrix/files-pipe/blob/main/src/lib/files/pipe.ts

Of all my benchmarks I stopped at piscina https://github.com/nikolaxhristov/bench-piscina, but if you have other suggestions, I'm open.

I have a repo that's a bit unmaintained at the moment to test those exact extensions https://github.com/nikolaxhristov/bench and the files-pipe component itself.

[kyeshmz]

Is there a way to cache previously compressed files? It would help out rather than making things parallel, as I think that is much harder

[NikolaRHristov]

@kyeshmz See: #49

[azat-io]

It would be nice to be able to run the compressor in parallel. I have a lot of images in my project too and the application build currently takes 26 minutes