Switch over logprobs to work with post-sampling #66
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bdashore3
force-pushed
the
better-logprobs
branch
from
9b20da8
to
36ea794
Compare
Previously, pre-sampling logprobs were used from the raw logits, but newer versions of exl2 allow for returning token probs post-sampling. Convert these to logprobs and send to the user. Signed-off-by: kingbri <[email protected]>
bdashore3
force-pushed
the
better-logprobs
branch
from
36ea794
to
8e07d20
Compare
Update to v0.0.13.post2 Signed-off-by: kingbri <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Change logits to return post-sampling vs pre-sampling to accurately represent the correct percentages.
Also switch the generator to use the streaming_ex gen which returns a more flexible dictionary.
Blocking on release of exllamav2 v0.0.13.post2