New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support microsoft contrib ONNX operator MatMulNBits #3390

Open

hgaspar opened this issue Aug 19, 2024 · 0 comments

Labels

UAI

hgaspar commented Aug 19, 2024 •

edited by gyulaz-htec

Loading

Such an operator appears in LLM models quantized to int4 (also with GroupQueryAttention nodes), via the genai tool.

Only N=4 needs to be supported in near term (i.e. 4 bits)

For reference, see the operator description in:

https://github.com/microsoft/onnxruntime/blob/main/docs/ContribOperators.md#com.microsoft.MatMulNBits

Here, supports means:

Ability to parse models that contain it.
Implement it via known operators.

hgaspar added the UAI label

gyulaz-htec changed the title ~~Support microsoft contib ONNX operator MatMulNBits~~ Support microsoft contrib ONNX operator MatMulNBits

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment