New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

用xinference无法加载4-bit的glm-4-9b-chat #596

Open

2 tasks

bunnieberrie opened this issue Oct 17, 2024 · 0 comments

Open

2 tasks

用xinference无法加载4-bit的glm-4-9b-chat #596

bunnieberrie opened this issue Oct 17, 2024 · 0 comments

bunnieberrie commented Oct 17, 2024

System Info / 系統信息

CUDA=12.4
transformers=4.44.2
torch=2.4.1+cu124
OS=windows 11 23H2
python=3.11.10

Who can help? / 谁可以帮助到您？

No response

Information / 问题信息

The official example scripts / 官方的示例脚本
My own modified scripts / 我自己修改的脚本和任务

Reproduction / 复现过程

使用xinference加载4-bit的glm4时提示如下

ValueError: [address=127.0.0.1:6898, pid=35832] Only 8-bit quantization is supported if it is not linux system or cuda device

Expected behavior / 期待表现

想了解一下是CUDA版本不对还是其他原因

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment