DataLoader Performance improvement

[gitttt-1234]

The primary bottleneck in our training pipeline is the dataloader performance (currently training time per epoch is very high due to the dataloader - IterDatapipe).

We should implement the PreLoader module (check SLEAP) in Sleap-NN and benchmark the training time. The training time should be optimized to improve/ match the current SLEAP performance.

Ref (current Torch issue): pytorch/data#1196

[talmo]

Profile!! --> https://lightning.ai/docs/pytorch/stable/tuning/profiler_basic.html

Break down the performance by steps, adding one block of the preloader at a time.

[gitttt-1234]

We compared and benchmarked the performance of IterDatapipes with LitData, and found that LitData is much faster and efficient for data processing. #80 addresses the plan for refactoring our current data pipeline.