try out libdeflate

[geospatial-jeff]

it might be faster than imagecodecs for deflate compression

https://github.com/dcwatson/deflate

[kylebarron]

The only downside is that libdeflate doesn't support streaming compression, but I'm guessing that's irrelevant for aiocogeo

[cgohlke]

The deflate Python package only handles libdeflate_gzip compression so far. For TIFF you'll need libdeflate_zlib.

A codec based on libdeflate will be included in the next release of imagecodecs. No ETA...

[kylebarron]

The deflate Python package only handles libdeflate_gzip compression so far. For TIFF you'll need libdeflate_zlib.

Is this as simple as replacing libdeflate_gzip_compress with libdeflate_zlib_compress? Would seem straightforward (if verbose) to copy the bindings again for zlib?

https://github.com/dcwatson/deflate/blob/f8572aa03e2ceab2fda85e83939691eb5d57bb5b/deflate.c#L24-L33

[kylebarron]

I made an issue in the deflate repo: dcwatson/deflate#2

[kylebarron]

From the linked issue:

For compression, yes, it would be pretty trivial to just use libdeflate's zlib API in the same way the gzip API is currently used. But for decompression, you need to allocate a buffer for libdeflate to write into. Gzip has the benefit of including the original decompressed size as a check at the end of an archive, so I can know exactly how much memory to allocate. For other decompression APIs, you would need to guess and potentially loop, or expose an optional buffer_size to the Python call (or both). It was just much simpler to only do gzip to start.

So it isn't completely trivial to add zlib decompression (at least for me, maybe trivial for a C programmer 😄 )