Writing larger-than-memory dataset

[wbthomason]

I have a collection of data with unknown length that is larger than memory on my system. Is there a way I can open a file/dataset in a streaming mode to incrementally write my data and not hold the entire dataset in memory?

[matz-e]

Hi, it seems like we're missing a good, easily accesible example for this. Definitely possible, as HDF5 has growing datasets when one uses chunking (but this needs to be set up at dataset creation time). In addition, with dataspaces, one can select only a subset of the datasets to write to, thus not needing to keep all data in memory at all times.

See this part of our unit tests for some basic usage of chunking. I hope that helps!

[wbthomason]

Thanks! I'd actually already figured out chunking/dataset resizing, but this seemed to keep all my data in memory. For writes, using chunking + selecting a region of the dataset explicitly is expected to limit the loaded data?

[matz-e]

Yes, I think you need to explicitly select a narrow region to write stuff to. IIRC, the process would roughly be as follows:

1. receive some data
2. resize dataset to accomodate fresh data
3. select a hyperslab at the end of the dataset with the freshly allocated chunk
4. write data with hyperslab to dataset, clear memory
5. go to 1.

[duchengyao]

It's possible to extend 2d array using h5easy?

I checked the code and it seems only supports extending an element but vector.

Is there some example for this task using h5easy?