data loading

[lucidrains]

what to use for data loading?

[lucidrains]

why are people voting for pytorch dataloading?? it is slow

[kobiso]

I haven't used FFCV but it looks nice, supporting webdataset format. It would be great if our data loader support something like webdataset as we need to handle large data.

[rom1504]

For the generator, I'd vote for webdataset for compatibility with existing datasets reasons
Be aware that generator training is slow so performance of the data loader doesn't impact training here. (But webdataset is fast)

For the prior, something like https://github.com/rom1504/embedding-reader makes sense imo: just use precomputed embeddings in npy files

[lucidrains]

🙏 amazing! i may include embedding-reader in the project at some point once i figure out a way to organize the training :) thank you!

[lucidrains]

i think i may go with ffcv.io

[rom1504]

I went through squirrel doc and I can't find a place where they describe what their format is like
The data storage format is the important part, because that's what makes it possible to read data in many libraries and frameworks
and that's also what make performance possible

that's the thing that is good with webdataset: it's just tar files of 1.jpg and 1.txt

[rom1504]

and ffcv sure why not but here are the important points of a data loading lib:

• it should work with all high latency storage easily (eg hdfs, s3, gcs)
• it should work with tensorflow, pytorch, jax
• it should be easy to read from and to write in

webdataset works for all these points
I'm not sure if it's the case for ffcv

[rom1504]

I don't see anything in ffcv doc about supporting non local filesystem
whereas webdataset supports it by supporting any command that can output to stdout (so eg it supports aws cli, gcs cli, hdfs cli, wget, ...)

[lucidrains]

ahh ok, i'll just go with webdatasets then, i trust you more at this point 😄

[rom1504]

I guess you may also be interested by pytorch/data#367
basically torch data now supports web dataset format https://pytorch.org/data/main/generated/torchdata.datapipes.iter.WebDataset.html#torchdata.datapipes.iter.WebDataset

I would however not advise to use torch data right now, webdataset support there has been merged 3 days ago, let's wait a little bit