Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix numeric instability in LayerNormalization #20257

Merged
merged 1 commit into from
Sep 13, 2024

Conversation

james77777778
Copy link
Contributor

When implementing SD3, I found that LayerNormalization is numeric unstable when dtype is set to float16.
This PR upcasts float16/bfloat16 to float32 for the computation.

I used backend.result_type(..., "float32") to handle higher precision dtypes, such as float64.

@codecov-commenter
Copy link

codecov-commenter commented Sep 13, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.35%. Comparing base (e7b5a5d) to head (39f29b7).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #20257      +/-   ##
==========================================
- Coverage   79.35%   79.35%   -0.01%     
==========================================
  Files         501      501              
  Lines       47642    47637       -5     
  Branches     8768     8765       -3     
==========================================
- Hits        37808    37803       -5     
  Misses       8054     8054              
  Partials     1780     1780              
Flag Coverage Δ
keras 79.20% <100.00%> (-0.01%) ⬇️
keras-jax 62.49% <100.00%> (-0.01%) ⬇️
keras-numpy 57.61% <100.00%> (+<0.01%) ⬆️
keras-tensorflow 63.88% <100.00%> (-0.01%) ⬇️
keras-torch 62.54% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@google-ml-butler google-ml-butler bot added kokoro:force-run ready to pull Ready to be merged into the codebase labels Sep 13, 2024
@fchollet fchollet merged commit 85e3681 into keras-team:master Sep 13, 2024
6 checks passed
@google-ml-butler google-ml-butler bot removed ready to pull Ready to be merged into the codebase kokoro:force-run labels Sep 13, 2024
@james77777778 james77777778 deleted the fix-numeric-instability branch September 15, 2024 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants