Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

resolve OOM errors in transfer-learning or warn that it needs a GPU #67

Open
charlesfrye opened this issue Sep 4, 2020 · 1 comment

Comments

@charlesfrye
Copy link
Collaborator

On my reasonably-equipped home cpu-machine and on the (cpu) hub, the transfer-learning example causes OOM errors -- sometimes even before getting to the model.fit call.

They're pretty scary-looking, if you're not expecting them, and depending on the exact system parameters, they can sometimes be triggered inside of the wandb.init call, making it look like our fault.

Potential solutions:

  • cut the dataset size in half
  • shrink the images to the minimum acceptable size for ResNet50 (80% of current size)
@charlesfrye
Copy link
Collaborator Author

It also doesn't help that there are two copies of the data: one normalized and one not normalized.

Moving the normalization into the data-loading or into the model (as is standard Keras practice these days, to prevent GPUs from getting starved) would help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant