-
Notifications
You must be signed in to change notification settings - Fork 446
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Disable the exllama on all non-cuda devices. #2003
Conversation
1. Disable the exllama on all non-cuda devices. 2. Don't raise the error when running on non-cuda device. Signed-off-by: yuanwu <[email protected]>
Signed-off-by: yuanwu <[email protected]>
Signed-off-by: yuanwu <[email protected]>
Signed-off-by: yuanwu <[email protected]>
@IlyasMoutawwakil , could you help review? This PR is to disable exllama on non-cuda devices, since other devices don't support it yet, this will enable devices like intel habana can work w/ autogptq. The one open is: do we need add rocm into SUPPORT_EXLLAMA_DEVICES, since the exllama says "it's theoretically supported but not validated", could you share your insights? |
@yao-matrix rocm is registered as cuda so that shouldn't be an issue, I think there's no need for a list of supported devices as exllama is written in cuda and will probably on work on cuda/hip/rocm machines, that should simplify the PR. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the PR @yuanwu2017 ! LGTM !
Co-authored-by: Ilyas Moutawwakil <[email protected]>
Co-authored-by: Ilyas Moutawwakil <[email protected]>
Co-authored-by: Ilyas Moutawwakil <[email protected]>
Co-authored-by: Ilyas Moutawwakil <[email protected]>
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
@yao-matrix can you run |
Signed-off-by: yuanwu <[email protected]>
@IlyasMoutawwakil The error is not related with the patch. Please help to merge it. |
What does this PR do?
Fixes # (issue)
Before submitting
Who can review?