does nequip support load/save data from/to hdfs?

[wenyan4work]

if the hdfs is not mounted as a normal file system, but can only be accessed using the commands 'hdfs dfs put/get', does nequip support this use-case?

[Linux-cpp-lisp]

Hi @wenyan4work ,

No, NequIP only supports loading from files that can be accessed through the filesystem. If I understand right, hdfs dfs get just copies down the file locally, so then you can just feed NequIP the local file.

[wenyan4work]

Thank you. I was thinking about things like this https://pytorch.org/blog/efficient-pytorch-io-library-for-large-datasets-many-files-many-gpus/ where the application may transparently load&save data with hdfs. Since nequip data is usually not that big, "hdfs dfs -get" should be sufficient.

[Linux-cpp-lisp]

Ah I see— you could certainly implement something like this as a subclass of AtomicInMemoryDataset (see dataset.py) but as the name suggests we currently assume that the entire dataset can be fit in RAM. While the code is set up to relax this assumption later and allow streaming datasets (AtomicDataset), some key features currently are not written to allow this (dataset.statistics) and we do not expect that to change in the near future.