Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add early support for torchdata.stateful_dataloader.StatefulDataLoader within the Accelerator #2895

Merged
merged 74 commits into from
Aug 22, 2024
Merged

Add early support for torchdata.stateful_dataloader.StatefulDataLoader within the Accelerator #2895

merged 74 commits into from
Aug 22, 2024

Commits on Jun 22, 2024

  1. temporary commit

    @byi8220
    byi8220 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    79a8fa2 View commit details
    Browse the repository at this point in the history

  2. checkout?

    @byi8220
    byi8220 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    efa1e7d View commit details
    Browse the repository at this point in the history

  3. dataloader wrapper

    @byi8220
    byi8220 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    8dc107d View commit details
    Browse the repository at this point in the history

  4. tmp

    @byi8220
    byi8220 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    f342f4c View commit details
    Browse the repository at this point in the history

  5. weird failing test

    @byi8220
    byi8220 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    065849a View commit details
    Browse the repository at this point in the history

  6. trying multiple inheritance

    @byi8220
    byi8220 committed Jun 22, 2024
    Configuration menu
    Copy the full SHA
    1e3fad1 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2024

  1. DataLoaderAdapter

    @byi8220
    byi8220 committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    a41cf38 View commit details
    Browse the repository at this point in the history

  2. make style

    @byi8220
    byi8220 committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    8831488 View commit details
    Browse the repository at this point in the history

  3. Some dark magic dynamic reflection (for backwards compat)

    @byi8220
    byi8220 committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    140f1e6 View commit details
    Browse the repository at this point in the history

  4. typo

    @byi8220
    byi8220 committed Jun 23, 2024
    Configuration menu
    Copy the full SHA
    727afeb View commit details
    Browse the repository at this point in the history

Commits on Jun 25, 2024

  1. some tests

    @byi8220
    byi8220 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    73683b4 View commit details
    Browse the repository at this point in the history

  2. more mixin stuff

    @byi8220
    byi8220 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    32c318e View commit details
    Browse the repository at this point in the history

  3. maybe found broken test?

    @byi8220
    byi8220 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    57c6f57 View commit details
    Browse the repository at this point in the history

  4. this is a very invasive feature

    @byi8220
    byi8220 committed Jun 25, 2024
    Configuration menu
    Copy the full SHA
    ed612d1 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2024

  1. i think the feature is done?

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    511050e View commit details
    Browse the repository at this point in the history

  2. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    8dbc1a3 View commit details
    Browse the repository at this point in the history

  3. add xpu support (huggingface#2864)

    @faaany @byi8220
    faaany authored and byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    df43960 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'stateful-dataloader' of https://github.com/byi8220/acce… 

    …lerate into stateful-dataloader
    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    c778e32 View commit details
    Browse the repository at this point in the history

  5. better tests

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    4e00055 View commit details
    Browse the repository at this point in the history

  6. discovered a bug

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    0471fe3 View commit details
    Browse the repository at this point in the history

  7. maybe fixed bug?

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    3036b7f View commit details
    Browse the repository at this point in the history

  8. make style

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    9ade2e9 View commit details
    Browse the repository at this point in the history

  9. hopefully this is PR ready

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    ba0f5c6 View commit details
    Browse the repository at this point in the history

  10. properly skip tests

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    b774291 View commit details
    Browse the repository at this point in the history

  11. parameterize

    @byi8220
    byi8220 committed Jun 26, 2024
    Configuration menu
    Copy the full SHA
    fde597d View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    f273abc View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    8a46eb6 View commit details
    Browse the repository at this point in the history

Commits on Jul 13, 2024

  1. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Jul 13, 2024
    Configuration menu
    Copy the full SHA
    e4e1cac View commit details
    Browse the repository at this point in the history

Commits on Jul 15, 2024

  1. temporary commit

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8bf2fe2 View commit details
    Browse the repository at this point in the history

  2. checkout?

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    ca4338d View commit details
    Browse the repository at this point in the history

  3. dataloader wrapper

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    c38f317 View commit details
    Browse the repository at this point in the history

  4. tmp

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    17a2a19 View commit details
    Browse the repository at this point in the history

  5. weird failing test

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    b39a606 View commit details
    Browse the repository at this point in the history

  6. trying multiple inheritance

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d1e82e0 View commit details
    Browse the repository at this point in the history

  7. DataLoaderAdapter

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d99d734 View commit details
    Browse the repository at this point in the history

  8. make style

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    39b2866 View commit details
    Browse the repository at this point in the history

  9. Some dark magic dynamic reflection (for backwards compat)

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f58f609 View commit details
    Browse the repository at this point in the history

  10. typo

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    f2119cf View commit details
    Browse the repository at this point in the history

  11. some tests

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    7adec94 View commit details
    Browse the repository at this point in the history

  12. more mixin stuff

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8850af3 View commit details
    Browse the repository at this point in the history

  13. maybe found broken test?

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6ff0f68 View commit details
    Browse the repository at this point in the history

  14. this is a very invasive feature

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    4f28d2e View commit details
    Browse the repository at this point in the history

  15. i think the feature is done?

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a9b637d View commit details
    Browse the repository at this point in the history

  16. better tests

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    0384543 View commit details
    Browse the repository at this point in the history

  17. discovered a bug

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    0e0515d View commit details
    Browse the repository at this point in the history

  18. maybe fixed bug?

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    809aca0 View commit details
    Browse the repository at this point in the history

  19. make style

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    5145c2d View commit details
    Browse the repository at this point in the history

  20. hopefully this is PR ready

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    ca74ff2 View commit details
    Browse the repository at this point in the history

  21. properly skip tests

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    a8f8bf3 View commit details
    Browse the repository at this point in the history

  22. parameterize

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    59738f4 View commit details
    Browse the repository at this point in the history

  23. Merge branch 'stateful-dataloader' of https://github.com/byi8220/acce… 

    …lerate into stateful-dataloader
    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    d264939 View commit details
    Browse the repository at this point in the history

  24. Update src/accelerate/utils/dataclasses.py 

    Co-authored-by: Zach Mueller <[email protected]>
    @byi8220 @muellerzr
    byi8220 and muellerzr committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8f04c1e View commit details
    Browse the repository at this point in the history

  25. Update src/accelerate/data_loader.py 

    Co-authored-by: Zach Mueller <[email protected]>
    @byi8220 @muellerzr
    byi8220 and muellerzr committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    45db4b9 View commit details
    Browse the repository at this point in the history

  26. merge conflicts

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    0ffc64b View commit details
    Browse the repository at this point in the history

  27. Merge branch 'stateful-dataloader' of https://github.com/byi8220/acce… 

    …lerate into stateful-dataloader
    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    03a7774 View commit details
    Browse the repository at this point in the history

  28. move imports

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    8d2c6c3 View commit details
    Browse the repository at this point in the history

  29. make style

    @byi8220
    byi8220 committed Jul 15, 2024
    Configuration menu
    Copy the full SHA
    6bfe871 View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2024

  1. merge conflicts?

    @byi8220
    byi8220 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    7a344e4 View commit details
    Browse the repository at this point in the history

  2. merges are breaking tests

    @byi8220
    byi8220 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    6ff997e View commit details
    Browse the repository at this point in the history

  3. fix test name

    @byi8220
    byi8220 committed Jul 17, 2024
    Configuration menu
    Copy the full SHA
    4739524 View commit details
    Browse the repository at this point in the history

Commits on Jul 22, 2024

  1. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Jul 22, 2024
    Configuration menu
    Copy the full SHA
    4de9159 View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2024

  1. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    abf815a View commit details
    Browse the repository at this point in the history

  2. Require safetensors>=0.4.3

    @byi8220
    byi8220 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    06597d4 View commit details
    Browse the repository at this point in the history

  3. undo last commit

    @byi8220
    byi8220 committed Jul 24, 2024
    Configuration menu
    Copy the full SHA
    4142c7f View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2024

  1. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    f02f18c View commit details
    Browse the repository at this point in the history

  2. minor style

    @byi8220
    byi8220 committed Jul 29, 2024
    Configuration menu
    Copy the full SHA
    35977ca View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge branch 'huggingface:main' into stateful-dataloader

    @byi8220
    byi8220 committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    597e910 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2024

  1. Merge branch 'main' into stateful-dataloader

    @byi8220
    byi8220 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    419f607 View commit details
    Browse the repository at this point in the history

  2. address pr comments

    @byi8220
    byi8220 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    4188d4c View commit details
    Browse the repository at this point in the history

  3. Torchdata version 0.8.0 is stable now

    @byi8220
    byi8220 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    51377a4 View commit details
    Browse the repository at this point in the history

  4. added docs and require torchdata>=0.8.0 for testing

    @byi8220
    byi8220 committed Aug 20, 2024
    Configuration menu
    Copy the full SHA
    f4b6bb5 View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2024

  1. test base_dataloader attr doesn't cause infinite recursion

    @byi8220
    byi8220 committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    d02dfcc View commit details
    Browse the repository at this point in the history

  2. address pr

    @byi8220
    byi8220 committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    21bc420 View commit details
    Browse the repository at this point in the history

  3. replace super().__iter__ with self.base_dataloader.__iter__

    @byi8220
    byi8220 committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    74e2f53 View commit details
    Browse the repository at this point in the history