RAG with Llama-cpp-python - parameters validation

[calypset]

Hello,

I am building a RAG with Llama-cpp-python and langchain LlamaCpp for a few hundred PDFs of scientific information and a few GPUs.

I have tried optimizing the parameters of the LLM to my best knowledge based on information online.

I was wondering if those parameters would seem appropriate for the intended purpose of interrogating a large set of data?

Loading the model as such with parameters:

llm = LlamaCpp(
        model_path=model_source,
        **params,
        chat_format='zephyr', 
        stop=['</s>'], 
        callbacks=callbacks
    )

Parameters loaded:

N_GPU_LAYERS  = -1
ROPE_FREQ_SCALE  = 2.5
ROPE_FREQ_BASE = 26000
N_DISCARD = 1
N_KEEP = 4
N_PARTS = 1
N_CTX = 16384
N_BATCH = 2048
N_PREDICT = -1
CONTEXT_WINDOW = 16384
MAX_TOKENS = 16384
MAX_NEW_TOKENS = 4096
LAST_N_TOKENS_SIZE  = 4096
TEMPERATURE = 1
REPEAT_PENALTY = 1
F16_KV = True
TOP_P = 1
TYPICAL_P = 0.9
MIN_P = 0.1
TFS = 0.999
VERBOSE = True
ECHO = False