Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor: Optimize batch shuffling implementation for better performance #252

Merged
merged 4 commits into from
Oct 30, 2024
Merged

refactor: Optimize batch shuffling implementation for better performance #252

merged 4 commits into from
Oct 30, 2024

Conversation

L-M-Sherlock
Copy link
Member

@L-M-Sherlock L-M-Sherlock commented Oct 30, 2024 •
edited
Loading

It's ~40% faster in small dataset. ~20% faster in large dataset.

This PR refactors the data loading and batch processing implementation with the following key changes:

  1. Simplified batch processing logic by removing complex BatchStrategy abstraction
  2. Optimized Tensor shape handling by using [1, pad_size] instead of unsqueeze()
  3. Redesigned data loader structure using BatchTensorDataset to pre-convert data to Tensors
  4. Simplified shuffle logic by randomizing at batch level directly

These changes are internal implementation optimizations that do not affect external APIs.

@L-M-Sherlock L-M-Sherlock added the enhancement New feature or request label Oct 30, 2024
@L-M-Sherlock L-M-Sherlock changed the title Improve performance refactor: Optimize batch shuffling implementation for better performance Oct 30, 2024
@L-M-Sherlock L-M-Sherlock marked this pull request as ready for review October 30, 2024 10:59
@asukaminato0721 asukaminato0721 merged commit 7477d2b into main Oct 30, 2024
3 checks passed
@asukaminato0721 asukaminato0721 deleted the improve-performance branch October 30, 2024 11:25
L-M-Sherlock added a commit to L-M-Sherlock/anki that referenced this pull request Oct 30, 2024
@L-M-Sherlock
Update to FSRS-rs v1.4.3
598d14c 
improve performance: open-spaced-repetition/fsrs-rs#252
dae pushed a commit to ankitects/anki that referenced this pull request Nov 5, 2024
@L-M-Sherlock
Feat/align FSRS-rs with PyTorch Implementation (#3540)
0fa9e0e 
* Feat/align FSRS-rs with PyTorch Implementation

* Update to FSRS-rs v1.4.2

fix https://forums.ankiweb.net/t/anki-24-10-release-candidate/51191/37?u=l.m.sherlock

* Update to FSRS-rs v1.4.3

improve performance: open-spaced-repetition/fsrs-rs#252
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@asukaminato0721 asukaminato0721 asukaminato0721 approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@L-M-Sherlock @asukaminato0721