Nexus File Chunk Sizes

[Modularius]

Variable sized vectors in HDF5 are stored as non-contiguous chunks of bytes. Chunks themselves are contiguous however, and must be read in full when they are written to. Chunk size should be optimized to balance the needs of reading and repeated writing. As a general rule, large chunk size is best for reading, whilst small chunk size is best for writing.

It must be determined how we should choose chunk size, should this be:

• Chosen dynamically when a NeXus file is first created (e.g. chunks set to 2x, 5x, or 10x the size of the first write).
• Set by command line arguments at runtime.
• Hard-coded into the file writer.
• The NeXus writer stored the maximum write size from previous runs and uses this to choose the chunk size.