Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add streaming tool use #1884

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

feat: add streaming tool use #1884

wants to merge 3 commits into from

Conversation

lsorber
Copy link
Contributor

@lsorber lsorber commented Dec 25, 2024

This PR upgrades the chatml-function-calling chat handler with support for streaming tool use and fixes #1883, #1869, and #1756, among other improvements.

Changes:

  1. General:
    a. ✨ If no system message is supplied, add an empty system message to hold the tool metadata.
    b. ✨ Add function descriptions to the system message so that tool use is better informed (fixes chatml-function-callling not adding tool description to the prompt. #1869).
    c. ✨ Replace print statements relating to JSON grammars with RuntimeWarning warnings.
    d. ✅ Add tests with fairly broad coverage of the different scenarios.
  2. Case "Tool choice by user":
    a. ✨ Add support for more than one function call by making this a special case of "Automatic tool choice" with a single tool (subsumes Support parallel function calls with tool_choice #1503).
  3. Case "Automatic tool choice -> respond with a message":
    a. ✨ Use user-defined stop and max_tokens.
    b. 🐛 Replace incorrect use of follow-up grammar with user-defined grammar.
  4. Case "Automatic tool choice -> one or more function calls":
    a. ✨ Add support for streaming the function calls (fixes Feature request: add support for streaming tool use #1883).
    b. ✨ Make tool calling more robust by giving the LLM an explicit way to terminate the tool calls by wrapping them in a <function_calls></function_calls> block.
    c. 🐛 Add missing ":" stop token to determine whether to continue with another tool call, which prevented parallel function calling (fixes chatml-function-calling chat format fails to generate multi calls to the same tool #1756).
    d. ✨ Set temperature=0 to determine whether to continue with another tool call, similar to the initial decision on whether to call a tool.

@lsorber
Copy link
Contributor Author

lsorber commented Dec 26, 2024

@abetlen The tests all pass, but the macOS ones were terminated after a timeout. I think this is because of a lack of CPU and or memory resources because the tests run fine on my macOS machine.

@SubatomicPlanets
Copy link

I would love to see this merged! Actually there are quite a lot of good pull requests here that i would like to see merged... But this one is top priority!

@lsorber
Copy link
Contributor Author

lsorber commented Jan 5, 2025

Update: I rebased on the latest main and included a few tiny improvements to further improve tool calling robustness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants