use fast softmax only on prefill #1159
Conversation
jaygala223
requested review from
mandy-li,
libinta and
dvarshney-habana
as code owners
|
@jaygala223 can you point to the source for this change? or is there an issue? From literature, fast softmax can be used in prefill and can be beneficial in any context where the softmax operation is a computational bottleneck. |
libinta
added
the
synapse1.17
|
If there is an accuracy issue raised by using fast softmax, can you point out to the issue, or bring it up here for the reference. It could be beneficial for other models and prevent others to root causing in similar cases. |
use fast softmax only on prefill huggingface#1159
use fast softmax only on prefill huggingface#1159
|
@yafshar apologies for the delayed response. There was a performance regression which got introduced in a patch revert for context manager. This PR fixes it |
@jaygala223 thanks, does this affect other models? Should we consider the same change for other models? |
|
@libinta please correct the PR label |
No, it does not affect other models |
|
Please run |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
use fast softmax only on prefill