Feedback about local LLM usage with paper-qa (share your experience about different LLMs and their parametrization).

[Snikch63200]

After a lot of trials with different local LLMs and different parameters (also called 'options'), I decided to open this discussion to share experiences about this.

My aim is to compare performances of different local models, with different configurations.

This is not a discussion about configuration issues but configuration optimization.
This is not, stricto sensu, a discussion about paper-qa settings

For each trial thanks to precise :

• Hardware (GPU type, CUDA CORES, VRAM, CPU type, CPU CORES, RAM)
• LLM serving type (Ollama, Llamafile, etc...)
• LLM model (llama3.1:70b, mixtral:8x7b, phi, etc...)
• Example (code snippet) of your configuration. Comments appreciated.
• Prompt used (with approximate tokens)
• PaperQA settings when not 'default', especially answer_max_sources, evidence_k and max_concurrent_requests that have a major impact on speed and answer relevance.
• A comment about answer quality.
• Request time.

New ideas for standardization of tests are welcome.

[Snikch63200]

As an example :

• Hardware : RTX 4090, 16 384 CUDA cores, 24 GB VRAM DDR6X, IntelCore I9-13900KF 3000 Mhz 24 cores 32 threads, 64 GB RAM DDR4
• serving type : ollama
• model : llama3.1:70b
• config :

local_llm_config = dict(
                        model_list=[
                                    dict(
                                        model_name=f"ollama/llama3.1:70b",
                                        litellm_params=dict(
                                                            model=f"ollama/llama3.1:70b",
                                                            api_type= "ollama",
                                                            api_base="http://192.168.1.15:11434",
                                                            cache= True,
                                                            mirostat=0,
                                                            mirostat_eta=0.1,
                                                            mirostat_tau=5.0,
                                                            num_ctx=32768, #Default 2048
                                                            repeat_last_n=64, # Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)
                                                            repeat_penalty=1.1,
                                                            temperature=0.2, #Low temperature to prevent consistency problems-default 0.7-
                                                            seed=0,
                                                            tfs_z=1,
                                                            num_predict=1000000,# Maximum number of tokens to predict when generating text. (Default: 128, -1 = infinite generation, -2 = fill context)
                                                            top_k=20, #Low top_k to prevent consistency problems-default 40-
                                                            top_p=0.9,
                                                            min_p=0.0
                                                            ),
                                        ),

                                    ]
                        )

• prompt "Quels sont les sites ayant livré des sépultures d'enfants, nouveaux nés ou périnataux enterrés dans des récipients en céramique (terrines, bassines, amphores, dolia). Répondez en Français." (about roman funeral archaelogy, in french - chose this question because it's very specific)
• answer_max_sources : 7, evidence_k : 20, max_concurrent_requests : 1
• The answer is globally relevant except 2 citations on 7 that concern ceramics buried near the corpse, not corpses buried inside ceramics -> LLM understanding problem.
• Request time : 0:35:56.063536