can't load 8B llava model

[end-me-please]

Describe the bug

Unable to load 8B llava model:
https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers

Is there an existing issue for this?

• I have searched the existing issues

Reproduction

1. apply fix from Bug fixes for llava multimodal #5038
2. try loading the model
3. AssertionError: Padding_idx must be within num_embeddings

Screenshot

No response

Logs

Traceback (most recent call last):
  File "D:\text-generation-webui\modules\ui_model_menu.py", line 249, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\modules\models.py", line 94, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\modules\models.py", line 363, in GPTQ_loader
    model = modules.GPTQ_loader.load_quantized(model_name)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\modules\GPTQ_loader.py", line 144, in load_quantized
    model = load_quant(str(path_to_model), str(pt_path), shared.args.wbits, shared.args.groupsize, kernel_switch_threshold=threshold)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\modules\GPTQ_loader.py", line 34, in _load_quant
    model = AutoModelForCausalLM.from_config(config, trust_remote_code=shared.args.trust_remote_code)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\auto\auto_factory.py", line 437, in from_config
    return model_class._from_config(config, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\installer_files\env\Lib\site-packages\transformers\modeling_utils.py", line 1401, in _from_config
    model = cls(config, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 1138, in __init__
    self.model = LlamaModel(config)
                 ^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\installer_files\env\Lib\site-packages\transformers\models\llama\modeling_llama.py", line 925, in __init__
    self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\text-generation-webui\installer_files\env\Lib\site-packages\torch\nn\modules\sparse.py", line 134, in __init__
    assert padding_idx < self.num_embeddings, 'Padding_idx must be within num_embeddings'
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: Padding_idx must be within num_embeddings

System Info

win10, CPU: 5700x, GPU: RTX 4070ti super

[Tedy50]

I think this multimodal pipeline is not supported yet
but it wud be nice to see it implemented.
GGUF format also already supports multimodal pipelines
exllama seems to be missing support yet

[Touch-Night]

If you use llama.cpp's multimodal support, you cannot have custom prompt template.