Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent agent-zero from overriding sampling parameters - or provide more options #272

Open
sammcj opened this issue Dec 20, 2024 · 0 comments

Comments

@sammcj
Copy link

sammcj commented Dec 20, 2024

At present Agent-Zero only has the basic 'temperature' sampling parameter and as it cannot be disabled - may mess up the more advanced sampling options configured in the model.

For context - temperature is a very basic sampling option which essentially injects noise (randomness) into part of the prediction algorithm and is very rudimentary and can be problematic for code generation (should pretty much always be set to 0 unless you're using min_p). Initially top_p replaced temperature as the most effective simple way of controlling "creativity" aka randomness, then min_p came out which is a good upgrade from top_p. In other words - we're two evolutions past setting 'temperature' alone.

It would be great if Agent Zero additionally let you set:

  • min_p
  • top_p
  • repetition penalty

and let you unset temperature (as in - don't send it with API requests to Ollama).

Further reading on why min_p is important huggingface/transformers#27670

Example of a sensible default configuration for a coding model (Qwen 2.5 Coder 32b in this case) that you don't want to be too ridged but want to retain high quality tokens:

### min_p sampling ##

# 1.0 disables top_p, so we can use min_p
PARAMETER top_p 1.0

# min_p works best with a small amount of temperature
PARAMETER temperature 0.2
PARAMETER min_p 0.9

PARAMETER repeat_penalty 1.05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant