stop dropping samples every batch

[Lewington-pitsos]

now each tokenization process keeps track of leftover samples and adds them back in when they accumulate to be long enough.

[norabelrose]

Sorry I'm sort of inclined not to merge this just for code simplicity reasons. Is there a reason you think it's really important not to drop samples ever?

[Lewington-pitsos]

The upshots I suppose are greater reproducibility and squeezing as much juice out of our dataset as we can (we end up dropping tens of thousands of samples for a reasonably sized run currently). Also gets more important if the user is forced to use a smaller batch size for some reason.

To me that is a very good trade in exchange for 20-30 lines of middlingly complicated code which don't touch anything else.

[dribnet]

Could you instead train for more than one epoch with different tokenization parameters to catch most of the leftovers?