best practice for snapshot_every_n_steps

[ShoufaChen]

Hello,

Thank you for your awesome implementation of StatefulDataloader.

I have a question about snapshot_every_n_steps. It seems there is not much detailed explanation about this argument.

• Will frequent snapshots (i.e., snapshot_every_n_steps=1) cause a data loading burden?
• What is the best practice for setting this value? Is it related to checkpointing frequency?

cc @andrewkho

[andrewkho]

Hi @ShoufaChen thanks for the issue, we should update the documentation to explain this better.

To answer your questions: it depends mainly on the size and composition of your state. If you're storing eg an int representing index or file-offset, then it shouldn't be an issue. If your state is very large, involving eg buffers of data for shuffling, then the overhead of creating a checkpoint and passing it through multiprocessing queue may slow down training, this variable lets you decrease the frequency of checkpointing. If eg you know you're checkpointing every 1000 steps, you can set this value to 1000.