CUDA Setup failed despite GPU being available. Please run the following command to get more information:

[WuRui-Ella]

配置：
系统：CentOS Linux release 7.9.2009
CUDA版本： 12.4
python版本： 3.11
NVIDIA-SMI ：550.54.14
我在使用hello_world推理时遇到了这个问题，已经提示重新编译了bitsandbytes，但还是出现了这个问题。

问题如下：
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone [email protected]:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=124
python setup.py install
CUDA SETUP: Setup Failed!
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone [email protected]:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=124
python setup.py install
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /home/ToolBench-master/toolbench/inference/qa_pipeline.py:13 in │
│ │
│ 10 os.environ['CUDA_VERSION'] = "12.4" │
│ 11 os.environ['NVCC'] = '/usr/local/cuda-12.4/bin/nvcc' │
│ 12 │
│ ❱ 13 from toolbench.inference.Downstream_tasks.rapidapi import pipeline_runn │
│ 14 │
│ 15 │
│ 16 if name == "main": │
│ │
│ /home/ToolBench-master/toolbench/inference/Downstream_tasks/rapidapi.py:1 │
│ 0 in │
│ │
│ 7 from termcolor import colored │
│ 8 import random │
│ 9 from toolbench.inference.LLM.chatgpt_function_model import ChatGPTFunc │
│ ❱ 10 from toolbench.inference.LLM.davinci_model import Davinci │
│ 11 from toolbench.inference.LLM.tool_llama_lora_model import ToolLLaMALoR │
│ 12 from toolbench.inference.LLM.tool_llama_model import ToolLLaMA │
│ 13 from toolbench.inference.LLM.retriever import ToolRetriever │
│ │
│ /home/ToolBench-master/toolbench/inference/LLM/davinci_model.py:9 in │
│ │
│ │
│ 6 import random │
│ 7 import openai │
│ 8 from typing import Optional │
│ ❱ 9 from toolbench.model.model_adapter import get_conversation_template │
│ 10 from toolbench.inference.utils import SimpleChatIO, react_parser │
│ 11 from toolbench.inference.Prompts.ReAct_prompts import FORMAT_INSTRUCTI │
│ 12 │
│ │
│ /home/ToolBench-master/toolbench/model/init.py:1 in │
│ │
│ ❱ 1 from toolbench.model.model_adapter import ( │
│ 2 │ load_model, │
│ 3 │ get_conversation_template, │
│ 4 │ add_model_args, │
│ │
│ /home/ToolBench-master/toolbench/model/model_adapter.py:21 in │
│ │
│ 18 │ AutoTokenizer, │
│ 19 │ LlamaForCausalLM, │
│ 20 ) │
│ ❱ 21 from peft import PeftModel │
│ 22 │
│ 23 from toolbench.tool_conversation import Conversation, get_conv_templat │
│ 24 from toolbench.model.compression import load_compress_model │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/init.py:2 │
│ 2 in │
│ │
│ 19 │
│ 20 version = "0.3.0" │
│ 21 │
│ ❱ 22 from .mapping import MODEL_TYPE_TO_PEFT_MODEL_MAPPING, PEFT_TYPE_TO_CON │
│ 23 from .peft_model import ( │
│ 24 │ PeftModel, │
│ 25 │ PeftModelForCausalLM, │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/mapping.py:16 │
│ in │
│ │
│ 13 # See the License for the specific language governing permissions and │
│ 14 # limitations under the License. │
│ 15 │
│ ❱ 16 from .peft_model import ( │
│ 17 │ PeftModel, │
│ 18 │ PeftModelForCausalLM, │
│ 19 │ PeftModelForSeq2SeqLM, │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/peft_model.py │
│ :31 in │
│ │
│ 28 from transformers.modeling_outputs import SequenceClassifierOutput, T │
│ 29 from transformers.utils import PushToHubMixin │
│ 30 │
│ ❱ 31 from .tuners import ( │
│ 32 │ AdaLoraModel, │
│ 33 │ AdaptionPromptModel, │
│ 34 │ LoraModel, │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/tuners/__init │
│ __.py:21 in │
│ │
│ 18 # limitations under the License. │
│ 19 │
│ 20 from .adaption_prompt import AdaptionPromptConfig, AdaptionPromptModel │
│ ❱ 21 from .lora import LoraConfig, LoraModel │
│ 22 from .adalora import AdaLoraConfig, AdaLoraModel │
│ 23 from .p_tuning import PromptEncoder, PromptEncoderConfig, PromptEncoder │
│ 24 from .prefix_tuning import PrefixEncoder, PrefixTuningConfig │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/peft/tuners/lora.p │
│ y:40 in │
│ │
│ 37 │
│ 38 │
│ 39 if is_bnb_available(): │
│ ❱ 40 │ import bitsandbytes as bnb │
│ 41 │
│ 42 │
│ 43 @DataClass │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │
│ 0.post2-py3.11.egg/bitsandbytes/init.py:7 in │
│ │
│ 4 # LICENSE file in the root directory of this source tree. │
│ 5 │
│ 6 from . import cuda_setup, utils │
│ ❱ 7 from .autograd._functions import ( │
│ 8 │ MatmulLtState, │
│ 9 │ bmm_cublas, │
│ 10 │ matmul, │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │
│ 0.post2-py3.11.egg/bitsandbytes/autograd/init.py:1 in │
│ │
│ ❱ 1 from ._functions import undo_layout, get_inverse_transform_indices │
│ 2 │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │
│ 0.post2-py3.11.egg/bitsandbytes/autograd/_functions.py:9 in │
│ │
│ 6 │
│ 7 import torch │
│ 8 │
│ ❱ 9 import bitsandbytes.functional as F │
│ 10 │
│ 11 │
│ 12 # math.prod not compatible with python < 3.8 │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │
│ 0.post2-py3.11.egg/bitsandbytes/functional.py:17 in │
│ │
│ 14 from typing import Tuple │
│ 15 from torch import Tensor │
│ 16 │
│ ❱ 17 from .cextension import COMPILED_WITH_CUDA, lib │
│ 18 │
│ 19 │
│ 20 # math.prod not compatible with python < 3.8 │
│ │
│ /home/anaconda3/envs/toolllm/lib/python3.11/site-packages/bitsandbytes-0.38. │
│ 0.post2-py3.11.egg/bitsandbytes/cextension.py:20 in │
│ │
│ 17 │ if lib is None and torch.cuda.is_available(): │
│ 18 │ │ CUDASetup.get_instance().generate_instructions() │
│ 19 │ │ CUDASetup.get_instance().print_log_stack() │
│ ❱ 20 │ │ raise RuntimeError(''' │
│ 21 │ │ CUDA Setup failed despite GPU being available. Please run the f │
│ 22 │ │ │
│ 23 │ │ python -m bitsandbytes │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError:
CUDA Setup failed despite GPU being available. Please run the following
command to get more information:

    python -m bitsandbytes

    Inspect the output of the command and see if you can locate CUDA

libraries. You might need to add them
to your LD_LIBRARY_PATH. If you suspect a bug, please take the
information from python -m bitsandbytes
and open an issue at: https://github.com/TimDettmers/bitsandbytes/issues
./scripts/inference_toolllama_helloworld_demo.sh:行3: --tool_root_dir: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行4: --backbone_model: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行5: --model_path: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行6: --max_observation_length: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行7: --observ_compress_method: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行8: --method: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行9: --input_query_file: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行10: --output_answer_file: 未找到命令
./scripts/inference_toolllama_helloworld_demo.sh:行11: --api_customization: 未找到命令

终端输入nvcc -V

能看出来cuda是正确安装的，

找不到解决办法了55555555，特来问问大家，有知道怎么解决的嘛

[Cgf601]

你好 请问你解决了吗