All data loaders support internal reading functions. I will use this issue to describe some Data loader and possible integration into dlio_benchmark.

Dali data loader

Examples: npz tfrecord

Suggestion: define input pipeline where we do the following: a) read files, b) extract samples, and c) resize. DaliReader will have a init, read, and finalize API.

TFRecord

Examples csv (experimental) and tfrecord

Suggestion Return tf.data.dataset which includes reading, extracting samples and resize. TensorflowReader will have a init, read, and finalize API.

Pytorch

The recommended way to use PyTorch is to define custom data loaders. But it has some custom image loading.

Suggested Changes

• I will create separate enums for TensorflowReaders and DaliReaders we support. They will have numbers similar to our ReaderType for compatibility.
• Rename our data loaders to DLIO_PYTORCH, DLIO_TENSORFLOW, and DLIO_DALI as this is our implementations.
• Similarly, rename our data reads as DLIO_CSV and so on.
• The new data loaders would be called NATIVE_TENSORFLOW and NATIVE_PYTORCH.
• For validation our current loaders work with our DLIOReaderType. If user selects The NATIVE_TENSORFLOW then it will be validated against TensorflowReaderType and similarly for DALI.
• The base classes for these reader would be different as well. We will have three baseclasses DaliBaseReader, DLIOBaseReader, PyTorchBaseReader, and TensorflowBaseReader.