Cannot deploy qwen2 gguf embedding model

[ensean]

LocalAI version:

• localai/localai:v2.20.1-cublas-cuda12-ffmpeg
  Environment, CPU architecture, OS, and Version:

Linux ip-172-31-24-204 6.5.0-1024-aws #24~22.04.1-Ubuntu SMP Thu Jul 18 10:43:12 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Describe the bug

Planning to deploy qwen2 embedding model, test with the flowing request:

{
    "input": "Your text string goes here",
    "model": "gte-qwen2-7b-instruct-gguf"
}

• model config v1

backend: sentencetransformers
name: gte-qwen2-7b-instruct-gguf
parameters:
  model: KeyurRamoliya/gte-Qwen2-7B-instruct-GGUF
embeddings: true

• error msg v1

{
    "error": {
        "code": 500,
        "message": "could not load model (no success): Unexpected err=OSError(\"KeyurRamoliya/gte-Qwen2-7B-instruct-GGUF does not appear to have a file named config.json. Checkout 'https://huggingface.co/KeyurRamoliya/gte-Qwen2-7B-instruct-GGUF/tree/main' for available files.\"), type(err)=<class 'OSError'>",
        "type": ""
    }
}

Ok, this seems to be some error about the hugging face model repo, so I downloaded the gguf file, placed it in the models directory

• model config v2

backend: sentencetransformers
name: gte-qwen2-7b-instruct-gguf
parameters:
  model: gte-qwen2-7b-instruct-q8_0.gguf
embeddings: true

• error message v2

{
    "error": {
        "code": 500,
        "message": "could not load model (no success): Unexpected err=OSError(\"sentence-transformers/gte-qwen2-7b-instruct-q8_0.gguf is not a local folder and is not a valid model identifier listed on 'https://huggingface.co/models'\\nIf this is a private repository, make sure to pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`\"), type(err)=<class 'OSError'>",
        "type": ""
    }
}

I have even created a folder sentence-transformers and move the gguf file into it, but the error is the same...

To Reproduce

Expected behavior

Logs

Additional context

[mudler]

@ensean did you tried installing any qwen model from the model gallery instead? You can also run any models in the gallery ( https://models.localai.io ) with:

local-ai run <model_name>

For instance to run qwen2:

local-ai run qwen2-7b-instruct

Check also: https://localai.io/docs/getting-started/models/#examples

[ensean]

Thanks a lot. The gallery only contains chat version of qwen2 LLM, and I have successfully deployed qwen2-7b-instruct to my server in the gallery. But I am planning to build a RAG system, so I need to deploy another embedding LLM. In fact, I can deploy the original LLM with the following config file, but I can not deploy the gguf version.

backend: sentencetransformers
name: gte-qwen2-7b-instruct
parameters:
  model: Alibaba-NLP/gte-Qwen2-7B-instruct
embeddings: true

[ensean]

It is fault to set the backend to sentencentransformers, the correct backend should be llama-cpp. The correct yaml should be

backend: llama-cpp
name: gte-qwen2-7b-instruct-gguf
parameters:
  model: gte-qwen2-7b-instruct-q8_0.gguf
embeddings: true