Relation of chunks and blocks

[Karol-G]

Hey,

I thoroughly read the documentation and all the blog posts on Blosc2, yet I still have some questions regarding the relationship between chunks and blocks. As I understand it, Blosc2 utilizes a superchunk (schunk) representing the entire array, which in turn consists of chunks, which themselves consist of blocks. My questions regarding this concept are as follows:

1. The idea behind introducing blocks, in addition to chunks, is to better utilize the L1/L2/L3 cache of the system, as described below, correct? Is there any other motivation behind the concept of blocks? Here is a section from the blog:

   Remember that having a second partition means we have better flexibility to fit the different partitions at the different CPU cache levels; typically, the first partition (aka chunks) should be made to fit in the L3 cache, whereas the second partition (aka blocks) should fit in the L2/L1 caches (depending on whether compression ratio or speed is desired).

2. In other file formats, such as Zarr, compression is performed at the chunk level, such that each chunk is compressed individually. However, in Blosc2, I assume compression is performed at the block level, such that each block is compressed individually, correct? Consequently, chunks themselves are not compressed in Blosc2, as it would not be possible to read only some of the blocks in a chunk without having to first decompress the entire chunk, correct?

3. If the above is correct, does that mean the compression ratio is solely determined by the block size, with the chunk size having no influence on the compression ratio?

4. Consequently, does that also mean that the chunk size only influences the read speed?

I hope these aren't too many questions, but I would be very thankful for any answers that help me understand Blosc2 better :)

Kind regards,
Karol

[FrancescAlted]

Your appreciations are mostly correct, in the sense that compression in Blosc2 only happens actually at block partition. Having said that, Zarr also uses Blosc by default (although only version 1), and the same applies to it, so not a big difference in how they work.

The main difference is that Blosc2 has the notion of multidimensionality for partitions for both blocks and chunks, while Blosc1 does not. That means that users have way more flexibility to adapt for their needs, for either achieving either better cratio or speed. Another difference is that Blosc2 has a masking feature to decompress only interesting blocks out of a chunk (see "Block Masks and Parallel I/O" slides in https://www.blosc.org/docs/Blosc2-WP7-LEAPS-Innov-2022.pdf).

But in general, yes, while only block size (shape for Blosc2) influences cratio, both chunk and block sizes do have an influence in compression speed due to caches in CPUs. Deciding which shapes are best is sometimes a tough (albeit interesting) task. But at least Blosc2 offers you a comprehensive set of parameters for helping during your experimentation (which you cannot magically replace by heuristics, unfortunately).

[Karol-G]

Thanks a lot for the answers and for providing additional information!