Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix] Handle out of range values in scalar quantization #2865

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

meiravgri
Copy link

@meiravgri meiravgri commented Jul 30, 2024

Issue description

This PR addresses an issue with out-of-range values during scalar quantization. Previously, when quantizing data, values outside the calibration range could result in incorrect quantization.

Solution

The PR fixes this issue by introducing a clip operation in the quantization process. This operation ensures that quantized values are correctly bounded within the expected range, thus preventing out-of-range values from producing incorrect results.

Example

calibration_set = np.array([[1,20,-3], [4,5,-60]], dtype=np.float32)
dataset = np.array([[-1,15,1]], dtype=np.float32). 
sq_embeddings = quantize_embeddings(
    dataset,
    precision="int8",
    calibration_embeddings=my_np_array,
)
print("quantizing dataset = ", sq_embeddings)

Result:
incorrect quantized values for the first and third entries, respectively, which are outside the calibration range.

quantizing dataset =  [[ -42   42 -112]]

After the PR:
The quantized values are correctly bounded and thus match the expected output.

quantizing dataset =  [[-128   42  127]]

@meiravgri
Copy link
Author

Hi @tomaarsen, could you please review my PR? Thanks!

@meiravgri meiravgri changed the title Handle out of range values in scalar quantization [fix] Handle out of range values in scalar quantization Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant