diff --git a/blogs/13_bfcl_v3_multi_turn.html b/blogs/13_bfcl_v3_multi_turn.html
index 2c28526e5..d8cf7d2ab 100644
--- a/blogs/13_bfcl_v3_multi_turn.html
+++ b/blogs/13_bfcl_v3_multi_turn.html
@@ -99,6 +99,67 @@
.code-toggle:hover {
text-decoration: underline;
}
+
+ .level-1 {
+ background-color: #E2E2E2;
+ /* Darkest gray for top level */
+ font-weight: bold;
+ }
+
+ .level-2 {
+ background-color: #E8E8E8;
+ /* Slightly lighter gray */
+ }
+
+ .level-3 {
+ background-color: #EFEFEF;
+ /* Medium gray */
+ }
+
+ .level-4 {
+ background-color: #F5F5F5;
+ /* Light gray */
+ }
+
+ .level-5 {
+ background-color: #FFFFFF;
+ /* Pure white */
+ }
+
+ .scoring-table {
+ width: 100%;
+ border-collapse: collapse;
+ table-layout: auto;
+ }
+
+ .scoring-table th,
+ .scoring-table td {
+ word-wrap: break-word;
+ padding: 5px;
+ text-align: center;
+ }
+
+ @media screen and (max-width: 768px) {
+
+ .scoring-table th,
+ .scoring-table td {
+ padding: 1px;
+ }
+ }
+
+ .note {
+ font-size: 0.8em;
+ color: #666;
+ }
+
+ .na {
+ color: #999;
+ font-style: italic;
+ }
+
+ .emoji {
+ display: block;
+ }
@@ -119,7 +180,7 @@ π¦ Gorilla: Large Langu
APIs
-
BFCL V3: Introducing Multi-Turn & Multi-Step Function Calling
+
BFCL V3: Multi-Turn & Multi-Step Function Calling
- Introduction
@@ -127,8 +188,7 @@ BFCL V3: Introducing Multi-Turn & Multi-Step Function Calling
- Existing Tool Calling Dataset
- Dataset Composition
- Data Curation
- - Data Validation
- - Model Inference & Execution
+ - Model Inference Process
- Evaluation Metrics
- Result & Error Analysis
-
@@ -195,100 +255,105 @@
- Last updated: 2024-12-04 Release date: 2024-09-19. Last updated: 2024-12-10. [Change
Log]
-
Introduction
+
Introduction
- The Berkeley Function-Calling Leaderboard (BFCL) V3 takes a significant leap forward by
- introducing a new multi-turn, and multi-step function calling (tool usage) category.
- Only at BFCL V3 β’ Multi-Turn & Multi-Step, you will see a LLM stuck in a loop, listing the current directory, write a non-existing
- file, and list the directory again... You will ask LLM to make a social media post.
- LLM will force you to spell your username and password to login despite the fact that you are already
- browsing other people's posts! This is only possible with multi-turn,
- and multi-step function calling (tool usage). Note that BFCL V3 contains the Expert Curated (Non-live) dataset introduced in BFCL V1 and User Contributed (Live) dataset introduced in BFCL V2 and the multi-turn, and multi-step category introduced in BFCL V3.
+ The Berkeley Function-Calling Leaderboard (BFCL) V3
+ makes a big step forward by introducing a new category for multi-turn and multi-step function calls (tool
+ use). Only in BFCL V3, you might see a model loop over and overβlisting a directory, trying to write to a
+ file that isn't there, and then listing again. Or it might demand your username and password even though
+ you've already logged in and are browsing other users' posts. Such scenarios are only possible when a model
+ can use multiple turns and steps in its function calls.
- Understanding these more advanced interactions builds on the foundation of single-turn single-step function calling, where models takes an user input prompt and selects one or more functions with appropriately filled parameters from a set of provided function options, without further interaction. If you're unfamiliar with single-turn single-step function calling and the evaluation metrics we used, check out our earlier blog on single-turn single-step function calling for a deeper dive.
-
-
+ Note that BFCL V3 still includes the Expert Curated (Non-live) dataset from BFCL V1 and
+ the User Contributed (Live) dataset from BFCL V2. On
+ top of that, it now tests how well models can handle back-and-forth (multi-turn) and step-by-step
+ (multi-step) interactions.
- BFCL V3 β’ Multi-Turn & Multi-Step is a critical advancement in evaluating how Large Language Models (LLMs) interact with diverse
- scenarios through invoking right functions.
- Multi-turn function calling allows models to engage in a back-and-forth interaction with users, making it
- possible for LLMs to navigate through
- the complex tasks by asking clarifying questions. In contrast to multi-turn (user t0, assistant t1,
- user t2, assistant t3, ..),
- multi-step is where the LLM can break the response down into multiple steps (user t0, assistant t1,
- assistant t2,..).
- This new paradigm mimics real-world agentic behaviors where AI assistants might have to plan execution
- paths, request and
- extract critical information, and handle sequential function invokes to complete a task.
+ If you're new to single-turn, single-step function calling, be sure to check out our earlier blog
+ post for more background. That post explains how a model should pick a single function and fill in its
+ parameters without asking follow-up questions.
-
- In addition, this is the first time BFCL performs API state verifications as the ground truth validation. In
- previous iterations, BFCL has been
- dominated by dissecting function parameter pairs using AST and matching them in a list of possible answers.
- In BFCL V3, we will not perform an exact match
- on the parameters but on the state. As long as the internal state of an API system(file system, travel
- booking system) stays intact, we mark them as correct.
+ With BFCL V3, we're looking at more complex tasks. Multi-turn function calls (user t0, assistant t1,
+ user t2, assistant t3, ..) let models interact with the user back-and-forth, asking questions and
+ refining their approach. Multi-step calls (user t0, assistant t1,
+ assistant t2,..) let a model break its response into smaller parts before giving a final answer.
+ This setup mimics real-world cases where an AI might need to plan, gather info, and chain several actions
+ together.
+
+
+ Another new twist in BFCL V3 is how we check the model's answers. Instead of dissecting function parameter
+ pairs using AST and matching them in a list of possible answers, we now verify the actual state of the API
+ system (like file systems or booking systems) after the model runs its functions. This gives us a more
+ realistic way to see if the model did the right thing.
Quick Links:
-
+
- In this blog, we start off by describing the difference between multi-step and multi-turn function calling
- and explaining why both concepts
- are important in the real world. We will then present the key features of our benchmarking and findings when
- evaluating against SOTA models.
- Lastly, we will showcase the evaluation dataset construction process and highlight the importance of
- a human annotated dataset.
-
+ In this post, we'll explain the difference between multi-step and multi-turn function calling and why both
+ matter in real use cases. Then we'll highlight what sets this new benchmark apart and share our findings
+ after testing top models. Finally, we'll walk through how we built the evaluation dataset and why having
+ human-annotated data is so important.
-
What is Multi-Step & Multi-Turn?
+
What is Multi-Step & Multi-Turn?
Single-Turn
In a single-turn interaction, assistant can fulfill a user's request by making one function call.
- The user's request is typically a straight forward instruction or question that is state-agnostic.
+ These requests are typically straightforward, self-contained, and state-agnostic (i.e., do not rely on
+ prior context).
+
Multi-Step
-
Multi-step refers to an interaction where the assistant performs several internal function calls to
- address a single user request. This can be interpreted as models proactively plan and gather information to fulfill a request.
+
+ Multi-step interactions require the assistant to execute multiple internal function calls to address a
+ single user request. This process reflects the assistant's ability to proactively plan and gather
+ information to deliver a comprehensive response. The user only interacts with the model once (at the
+ very beginning), and the model then interacts with the system back-and-forth to complete the task.
+
Multi-Turn
-
Multi-turn interaction involves multiple exchanges between the user and the assistant. Each turn can
- contain multiple steps. Multi-turn interaction requires models to collect from previous context to answer follow-up questions.
+
+ Multi-turn interactions involve an extended exchange between the user and the assistant, consisting of
+ multiple conversational turns. Each turn may involve several steps, and the assistant must retain and
+ utilize contextual information from previous exchanges to handle follow-up queries effectively. The user
+ will interact with the model multiple times throughout the process.
+
Why Multi-Turn Matters
-
Existing Tool Calling Dataset
+ Existing Tool Calling Dataset
Table Example
@@ -417,74 +482,77 @@ Existing Tool Calling Dataset
-
Dataset Composition
+
Dataset Composition
+
Newly Introduced Categories
- Berkeley Function-Calling
- Leaderboard (BFCL V3 β’ Multi-Turn & Multi-Step Function Calling) Data Composition
+ BFCL V3 β’ Multi-Turn & Multi-Step Function Calling Data Composition
-
Base Multi-Turn (200): This category covers the foundational yet sufficiently diverse
- basic multi-turn interactions. In this category, we provide complete information to call each
- function (either through current turn question, execution result from previous turn, or initial state
- configuration)
+ basic multi-turn interactions. In this category, we provide all the necessary information--whether from
+ the user request message, execution result from previous turn, or the output of exploratory functions--to
+ complete the task. The model is expected to handle the multi-turn interaction without any ambiguity.
-
- Augmented Multi-Turn (800): This category introduce additional complexity, such as
+ Augmented Multi-Turn (800): These categories introduce additional complexity, such as
ambiguous prompts or situations
- where the model must process multiple pieces of information across turns (similar to Multihop QA),
- requiring models to
- handle more nuanced decision-making, disambiguation, and conditional logic across multiple turns.
-
- Missing Parameters (200): This dataset challenges the model to identify required
- missing information that cannot be retrieved
- elsewhere in the system. In this scenario, we expect the LLM to ask for a follow-up to clarify the
- misinformation.
- This is distinct from certain entries in the Core Multi-Turn dataset where the question has implicit
- intent that can be
- answered by referencing the backend system.
-
-
- Missing Functions (200): This scenario denotes when we expect the model to recognize
- that no action
- should be taken given the lack of functions provided. If the LLM raises that concern, we then supply it
- with the hold-out
- functions that can successfully perform user intended tasks. Note that the Core dataset and the Missing
- Function dataset
- essentially contains the same sequence of actions except for the latter we hold-out a subset of
- functions on execution path to
- further challenge the model's inference ability.
-
-
- Long-Context Multi-Turn (200): This dataset challenges the model's resilience in long
- context scenarios on function
- calling. We inject random objects (e.g. hundreds of files in one directory or thousands of booking
- records) to mimic real world API output,
- which tend to be overtly informative. Here, we aim to test the model's ability to grasp the core
- information from an overwhelmingly large context.
-
+ where the model must process multiple pieces of information across turns (similar to Multihop QA).
+ Models must demonstrate more nuanced decision-making, disambiguation, and conditional logic across
+ multiple turns.
+ The following four sub-categories are included in the Augmented Multi-Turn dataset:
- Composite (200): Composite Category seeks to combine all three scenarios above to
- create an exceptionally
- hard challenge that, despite being rare, is important to handle when using autonomous agents at scale.
- Through this category,
- we want to convince the audience that a good model performance in this category offers a strong signal
- that LLMs can function as autonomous
- agents at scale despite rare and extremely difficult scenarios.
+ -
+
+
+ Missing Parameters (200): Tests the model's ability to recognize when essential
+ information is missing from the user request and cannot be inferred from the system. In these cases,
+ the
+ model should
+ request clarification rather than making unwarranted assumptions. This differs from cases in
+ the Base Multi-Turn set, where implicit intentions can be resolved by referencing the backend system.
+
+ -
+
+ Missing Functions (200): Requires the model to identify that no available function
+ can fulfill the user request. Once the model indicates this issue, we provide the missing functions in
+ the next turn.
+ Compared to the Base dataset, this scenario withholds a subset of functions at the beginning,
+ challenging the model
+ to infer the need for additional functions.
+
+ -
+ Long-Context Multi-Turn (200): Challenges the model's ability to maintain accuracy
+ and relevance in lengthy, information-dense contexts. We introduce large volumes of extraneous data
+ (e.g., hundreds of files or thousands of records) to test how well the model can extract crucial
+ details from an overwhelming array of information.
+
+ -
+ Composite (200): Combines all three augmented challengesβMissing Parameters,
+ Missing Functions, and Long-Contextβinto a single, highly complex scenario. While rare, success here
+ strongly indicates that the model can function effectively as an autonomous agent at scale, even in
+ intricate and demanding conditions.
+
+
+
+ style="max-width: 100%; width: 100%; margin-bottom: 20px;">
+
+ An example of how the missing parameter and missing function are augmented from the base entries.
+
+
Here we visualize the data statistics of the BFCL V3 Base Multi Turn dataset (the augmented
categories follow similar distributions):
@@ -511,13 +579,271 @@
Dataset Composition
Full Leaderboard Score Composition
+ The full bfcl v3 leaderboard score is composed of the following components. Number in the parenthesis
+ indicates the number of entries in each category.
+
+
+
+
+ |
+ π Overall Score
+ Unweighted Average
+ (4751)
+ |
+
+
+
+
+ |
+ π Non-Live (Single-Turn)
+ Unweighted Average
+ (1700)
+ |
+
+ π‘ Live (Single-Turn)
+ Weighted Average
+ (2251)
+ |
+
+ π Multi-Turn
+ Unweighted Average
+ (800)
+ |
+
+
+
+
+
+ |
+ π³ AST
+ Unweighted Average
+ (1150)
+ |
+
+ π Exec
+ Unweighted Average
+ (310)
+ |
+
+ π§ Relevance Detection
+ (240)
+ |
+
+
+
+ π³ AST
+ Weighted Average
+ (1351)
+ |
+
+ π§ Relevance Detection
+ Independent Categories
+ (900)
+ |
+
+
+
+ π Base Case
+ (200)
+ |
+
+ π Augmented Entries
+ Independent Categories
+ (600)
+ |
+
+
+
+
+
+ |
+ π€ Simple
+ Unweighted Average
+ |
+
+ π¦ Multiple
+ |
+
+ π Parallel
+ |
+
+ π¦π Parallel Multiple
+ |
+
+
+
+ π€ Simple
+ Unweighted Average
+ |
+
+ π¦ Multiple
+ |
+
+ π Parallel
+ |
+
+ π¦π Parallel Multiple
+ |
+
+
+
+
+ |
+
+
+
+ π€ Simple
+ |
+
+ π¦ Multiple
+ |
+
+ π Parallel
+ |
+
+ π¦π Parallel Multiple
+ |
+
+
+
+
+ |
+
+
+ |
+
+
+
+
+ |
+
+
+ |
+
+
+ |
+
+
+ |
+
+
+
+
+
+ |
+ π Python Simple AST
+ (400)
+ |
+
+ β Java Simple AST
+ (100)
+ |
+
+ π» JavaScript Simple AST
+ (50)
+ |
+
+
+
+ π Python Multiple AST
+ (200)
+ |
+
+ π Python Parallel AST
+ (200)
+ |
+
+ π Python Parallel Multiple AST
+ (200)
+ |
+
+
+
+ π Python Simple Exec
+ (100)
+ |
+
+ π REST API Exec
+ (70)
+ |
+
+
+
+ π Python Multiple Exec
+ (50)
+ |
+
+ π Python Parallel Exec
+ (50)
+ |
+
+ π Python Parallel Multiple Exec
+ (40)
+ |
+
+
+
+ β Irrelevance
+ (240)
+ |
+
+
+
+ π Python Live Simple AST
+ (258)
+ |
+
+ π Python Live Multiple AST
+ (1053)
+ |
+
+ π Python Live Parallel AST
+ (16)
+ |
+
+ π Python Live Parallel Multiple AST
+ (24)
+ |
+
+
+
+ β
Live Relevance
+ (18)
+ |
+
+ β Live Irrelevance
+ (882)
+ |
+
+
+
+ π Python Multi Turn Base
+ (200)
+ |
+
+ π Python Multi Turn Missing Function
+ (200)
+ |
+
+ β οΈ Python Multi Turn Missing Parameter
+ (200)
+ |
+
+ π Python Multi Turn Long Context
+ (200)
+ |
+
+
Data Curation Methodology
- In this section, we detail our data curation methodology for the BFCL V3 β’ Multi-Turn & Multi-Step dataset. The dataset curation
- process consists of hand-curated data generation for four components of BFCL V3 β’ Multi-Turn & Multi-Step: API codebase creation, graph
+
+
Data Curation Methodology
+ In this section, we detail our data curation methodology for the BFCL V3 β’ Multi-Turn & Multi-Step
+ dataset. The dataset curation
+ process consists of hand-curated data generation for four components of BFCL V3 β’ Multi-Turn &
+ Multi-Step: API codebase creation, graph
edge construction, task generation, and human-labeled ground truth multi-turn trajectories, as well as a
comprehensive data validation process.
Dataset with human-in-the-loop pre-processing and post-processing
@@ -528,8 +854,10 @@ Dataset with human-in-the-loop pre-processing and post-processing
execution results based on initial configurations.
- Our team believes that synthetic dataset by itself alone is not enough and human labeling is essential. We take care of the APIs created by humans as we believe human can
- generate more connected and densely important functions useful for evaluation purposes. Even with this, we went through 11 rounds of data-filtering, highlighting the importance and challenges of function calling.
+ Our team believes that synthetic dataset by itself alone is not enough and human labeling is essential. We
+ take care of the APIs created by humans as we believe human can
+ generate more connected and densely important functions useful for evaluation purposes. Even with this, we
+ went through 11 rounds of data-filtering, highlighting the importance and challenges of function calling.
1. API Codebase Creation
Primary Domain APIs:
- Vehicle Control:
startEngine(...), displayCarStatus(...),
- estimate_distance(...)
+ estimate_distance(...)
+
- Trading Bots:
get_stock_info(...), place_order(...),
- get_watchlist(...)
+ get_watchlist(...)
+
- Travel Booking:
book_flight(...),
- get_nearest_airport_by_city(...), purchase_insurance(...)
+ get_nearest_airport_by_city(...), purchase_insurance(...)
+
- Gorilla File System:
ls(...), cd(...), cat(...)
@@ -556,19 +887,25 @@ 1. API Codebase Creation
Cross-functional APIs:
- Message API:
send_message(...), delete_message(...),
- view_messages_received(...)
+ view_messages_received(...)
+
- Twitter API:
post_tweet(...), retweet(...),
- comment(...)
+ comment(...)
+
- Ticket API:
create_ticket(...), get_ticket(...),
- close_ticket(...)
+ close_ticket(...)
+
- Math API:
logarithm(...), mean(...),
- standard_deviation(...)
+ standard_deviation(...)
+
All eight domains took inspiration from our experience with Open Functions data collection and public
interest in popular agent application domains.
- The four primary API domains are evenly distributed across the test cases in Base, and Augmented Multi-Turn. For example, there are 200 test entries in
- Base category and 0-49 utilizes Gorilla File System, 50-99 utilizes Vehicle Control, 100-149 utilizes Trading Bots, and 150-199 utilizes Travel Booking.
+
The four primary API domains are evenly distributed across the test cases in Base, and Augmented
+ Multi-Turn. For example, there are 200 test entries in
+ Base category and 0-49 utilizes Gorilla File System, 50-99 utilizes Vehicle Control, 100-149 utilizes
+ Trading Bots, and 150-199 utilizes Travel Booking.
@@ -642,6 +979,7 @@ 4. Human-Labeled Ground Truth Multi-Turn Trajectories
Validation Process
+
@@ -661,7 +999,8 @@
1. Question Validation (π§βπ»)
Clarity and Specificity: Ambiguous questions are refined to provide more specific
instructions.
Example: A question like βUpload the documentβ is improved to βUpload the
- <NUMBER_OF_BYTES>.pdf document.β
+
<NUMBER_OF_BYTES>.pdf document.β
+
Complete Information: The question and previous questions up to the current turn or
through exploration within the environment must contain all the necessary details for the model to
@@ -680,20 +1019,23 @@ 2. Human-Labeled Ground Truth Validation (π§βπ»+ π» )
Executability: Ensuring the labeled functions can be correctly executed in sequence
without errors.
Example: If the question asks for booking a flight, the ground truth should correctly call the
- book_flight function without any internal errors.
+
book_flight function without any internal errors.
+
Alignment with User Request: The execution path must be reasonable and consistent
with the question's intent, whether implicit or explicit.
Example: If the question asks for a flight reservation without providing the flight's cost, the
implicit request is to fetch the current ticket price by calling get_flight_cost before
- book_flight.
+
book_flight.
+
Brevity: The execution path should be logically concise under the premise of the
previous two criteria.
Example: If the question asks for posting a tweet and mentioning another user, only the functions
authenticate(username, password) and post_tweet(content, tags, mentions)
should be called. The function mention(tweet_id, mentioned_usernames) is unnecessary since
- post_tweet can handle user mentions.
+
post_tweet can handle user mentions.
+
3. Initial Configuration Validation ( π» )
@@ -717,7 +1059,8 @@
4. Function List Validation ( π» )
Completeness: All necessary functions must be present in the function list, allowing
the model to make appropriate calls.
Example: If the task involves sending a tweet, the function list should include the
- post_tweet function to ensure the model can complete the action.
+
post_tweet function to ensure the model can complete the action.
+
5. API Code Validation (π§βπ»+ π» )
@@ -740,7 +1083,8 @@
5. API Code Validation (π§βπ»+ π» )
Automated Format Checkers: Tools like mypy (a static type checker) and
pydocstyle are used to enforce strict compliance with type hints and docstring formatting.
- These automated tools check for:
+ These automated tools check for:
+
- Type Consistency: Ensures that each function adheres to the expected types defined
by PEP484.
@@ -751,7 +1095,7 @@ 5. API Code Validation (π§βπ»+ π» )
-
Model Inference & Execution
+
Model Inference Process
Initialization
Each entry comes with its own initial_config, which is used to initialize the API backend
state. For example, a file system backend might start with a set of pre-existing files, and a messaging API
@@ -772,9 +1116,11 @@
Step Execution
the chat history so far.
Function Call Execution: If the model makes any valid function calls (i.e.,
decode_exec in the model handler returns a non-empty list), the function calls are executed
- in the API backend in the order that model provides.
+ in the API backend in the order that model provides.
+
Updating Chat History: The model's response is then added to the chat history, along
- with the execution results (if any). For prompting models, since they don't use the `tool` role tag, we will provide back the execution results under the `user` role.
+ with the execution results (if any). For prompting models, since they don't use the `tool` role tag, we
+ will provide back the execution results under the `user` role.
End of a Step
After updating the chat history with the model's response and execution results, the current step ends.
@@ -786,7 +1132,8 @@
Termination of a Turn
No Output Termination: If the model doesn't output any valid function calls in a step,
we consider this the end of the current turn and move on to the next turn. This could occur if the model
outputs chatting messages or if its response cannot be properly decoded into executable function calls
- (the latter usually happens when the model is in prompting mode and is not following instructions). This method aligns with how we
+ (the latter usually happens when the model is in prompting mode and is not following instructions). This
+ method aligns with how we
determine if the model makes any function call in the irrelevance category in the single-turn
scenario, and we find it to work effectively.
Step Limit Force Termination: If the model takes more than 20 steps within a turn, the
@@ -800,14 +1147,16 @@ Termination of a Turn
Note For Multi Turn Missing Function Category
In this category, one or more functions are held out from the function list provided to the model at the
- beginning; they will be provided in a later turns (never the first turn). For FC models, the added functions will
+ beginning; they will be provided in a later turns (never the first turn). For FC models, the added functions
+ will
just be appended to the tools list. But for prompting models, since we supplied all the tools
at the beginning in the system prompt and it's not possible to modify the system prompt in the middle of the
conversation, we will provide the held-out function definitions in the content of a user message instead.
Why We Avoid Certain Techniques (e.g. ReAct)
- In BFCL V3 β’ Multi-Turn & Multi-Step, we deliberately avoid using techniques like prompt engineering and ReAct, which combines
+
In BFCL V3 β’ Multi-Turn & Multi-Step, we deliberately avoid using techniques like prompt engineering
+ and ReAct, which combines
reasoning and acting through specific prompting methods. While ReAct and other techniques can improve
models' function calling performance in certain cases, we chose not to use it throughout the BFCL series
to evaluate base LLMs with the same standards to isolate the effects from using additional optimization
@@ -815,26 +1164,51 @@
Why We Avoid Certain Techniques (e.g. ReAct)
+
Evaluation Metrics
+
BFCL V3 has different evaluation metrics for single-turn and multi-turn tasks.
+
Single-turn Evaluation Metrics
+
+ Single-turn evaluation metrics (i.e., Abstract Syntax Tree (AST) Evaluation, Executable Function Evaluation, and Relevance Detection) are the same as those used in BFCL v1 and
+ BFCL v2. Please refer to the previous blog posts for more
+ details.
+
Multi-turn Evaluation Metrics
-
In BFCL V3 β’ Multi-Turn & Multi-Step, we employed state-based evaluation and response-based evaluation to assess the model's performance the multi-turn categories.
- At the end of
every turn, we mark an entry as correct if it passes both checks in all turns. Note that force-terminated entries will be marked wrong, even if they pass the checks.
-
+
For multi-turn tasks, two types of checks are used: state-based evaluation and
+ response-based evaluation.
+
+ These two checks are run at the end of every turn.
+ An entry is marked as correct if it passes
both checks in
all turns.
+ Note that force-terminated entries will be marked wrong, even if they happen to pass the checks.
-
- State-based evaluation focuses on comparing the backend system's state (excluding the private attributes, i.e., the ones that start with _) after all function calls are executed at the end of each turn of the conversation. We expect that given a user request, there can be multiple ways to fulfill the demand, which we are not able to measure, but the end state, or end result, should be consistent with ground truth labelling. The state-based evaluation capture the correctness of model executions that modify the internal state via write and delete e.g. create a new file, delete a stock from watchlist.
+ State-based evaluation focuses on comparing the backend system's state (excluding the
+ private attributes, i.e., the ones that start with _) after all function calls are executed
+ at the end of each turn of the conversation. We expect that given a user request, there can be multiple
+ ways to fulfill the demand, which we are not able to measure, but the end state, or end result, should
+ be consistent with ground truth labelling. The state-based evaluation capture the
+ correctness of model executions that modify the internal state via write and delete
+ e.g. create a new file, delete a stock from watchlist.
+
-
-
Response-based evaluation compares the model's execution path against the minimial viable execution result paths as labeled in ground truth. The minimial viable execution result paths refer to a list of function calls that must be executed in order to produce desired response as user requests. Having response-based evaluation ensure read only request can be properly evaluated e.g. get the stock price, get the weather information.
+
Response-based evaluation compares the model's execution path against the minimial
+ viable execution result paths as labeled in ground truth. The minimial viable execution result paths
+ refer to a list of function calls that must be executed in order to produce desired response as user
+ requests. Having response-based evaluation ensure read only request
+ can be properly evaluated e.g. get the stock price, get the weather information.
In the
- following sections, we will discuss the advantages and limitations of state-based evaluation in multi-turn function calling and why we need a subset-matched response-based evaluation as well.
+ following sections, we will discuss the advantages and limitations of state-based evaluation in multi-turn
+ function calling and why we need a subset-matched response-based evaluation as well.
Why State-based Evaluation
-
Minicking state offer a different perspective of real world performance evalution as autonomous agents can detour on its own discreet while achieving the tasks after all. Instead of only checking if each individual function output is correct, we
+
Minicking state offer a different perspective of real world performance evalution as autonomous agents can
+ detour on its own discreet while achieving the tasks after all. Instead of only checking if each individual
+ function output is correct, we
compare the attributes of the system's state after every turn against the expected state. If the model
successfully brings the system to the correct state at the end of each turn, it passes the evaluation.
For example, if a model is tasked with a series of actions such as:
@@ -846,17 +1220,29 @@
Why State-based Evaluation
In state-based evaluation, the system checks after each turn whether the file exists, whether the correct
data was written, and whether the file is properly closed. If all the required state attributes are
present and correct at each turn, the evaluation succeeds.
-
+
Limitations of State-Based Evaluation
- While state-based evaluation is a powerful tool for assessing multi-turn function calling models, it does have some limitations. For example, some functions don't have a direct impact on the system's state, such as get_zipcode_by_city or estimate_distance. We will not be able to tell if the model has actually invoked those functions or not, if relying solely on state-based evaluation. We want to make sure that the model is making the necessary function calls and reasoning through the task, instead of just memorizing or guessing the correct information; we want the model to call get_zipcode_by_city("Berkeley") to get the zip code for Berkeley is 94710, and then use that zip code to call get_weather_by_zipcode(94710) to get the weather information, instead of directly calling get_weather_by_zipcode(94710) and hope that it is the correct zip code for Berkeley (this would be hallucination!).
- In such cases, response-based evaluation can be a good complement to state-based evaluation, as it can provide additional insights into the model's behavior and decision-making process.
+ While state-based evaluation is a powerful tool for assessing multi-turn function calling models, it does
+ have some limitations. For example, some functions don't have a direct impact on the system's state, such as
+ get_zipcode_by_city or estimate_distance. We will not be able to tell if the model
+ has actually invoked those functions or not, if relying solely on state-based evaluation. We want to make
+ sure that the model is making the necessary function calls and reasoning through the task, instead of just
+ memorizing or guessing the correct information; we want the model to call
+ get_zipcode_by_city("Berkeley") to get the zip code for Berkeley is 94710, and then use that
+ zip code to call get_weather_by_zipcode(94710) to get the weather information, instead of
+ directly calling get_weather_by_zipcode(94710) and hope that it is the correct zip code for
+ Berkeley (this would be hallucination!).
+ In such cases, response-based evaluation can be a good complement to state-based evaluation, as it can
+ provide additional insights into the model's behavior and decision-making process.
Why Subset-Matched Response-based Evaluation
-
In earlier versions like BFCL V1 and V2, a pure response-based evaluation was used; the model response must match the ground truth in full. This approach evaluated the
+
In earlier versions like BFCL V1 and V2, a pure response-based evaluation was used; the model response must
+ match the ground truth in full. This approach evaluated the
model based on the immediate function response, either by analyzing the return values or by checking the
- Abstract Syntax Tree (AST) structure. However, it faces several limitations when it comes to multi-turn categories:
+ Abstract Syntax Tree (AST) structure. However, it faces several limitations when it comes to multi-turn
+ categories:
- Inconsistent Trajectories: In multi-turn tasks, models may take different, equally
valid trajectories that are hard to predict or constrain via the prompt. For instance, a model might
@@ -882,9 +1268,9 @@
Why Subset-Matched Response-based Evaluation
Example trajectory in multi-turn function calling that would fail in response-based evaluation but
succeed in state-based evaluation.
-
- The question asks about purchasing Nvidia stock, and in order to do this, relevant stock information like
+
+
In the above example, the question asks about purchasing Nvidia stock, and in order to do this, relevant stock information like
its current price must be retrieved. In the ground truth, the get_stock_info function is
invoked, with the stock symbol value (provided in the question) passed in. However, the model does not
know that the value in the question is indeed the stock symbol, so it tries to first retrieve the symbol
@@ -892,17 +1278,22 @@
Why Subset-Matched Response-based Evaluation
symbols and pattern-match the correct one. Although the model took more turns (function calls) and went
through a trial-and-error process, it ultimately achieved the goal.
This example illustrates how evaluating based solely on the response is not ideal. Sometimes, the model
- may not have enough information to determine the minimial viable execution result paths, and taking extra steps should not be
- penalized. Thus, we introduce a subset-matched response-based evaluation to address these limitations. The ground truth is the the minimial viable execution result paths labled with full context of the existing states. The model result is considered correct if it contains the ground truth as a subset, even if it contains additional function calls or takes a different trajectory.
+ may not have enough information to determine the minimial viable execution result paths, and taking extra
+ steps should not be
+ penalized. Thus, we introduce a subset-matched response-based evaluation to address these limitations. The
+ ground truth is the the minimial viable execution result paths labled with full context of the existing
+ states. The model result is considered correct if it contains the ground truth as a subset, even if it
+ contains additional function calls or takes a different trajectory.
Result & Error Analysis
+ Result & Error Analysis
-
Scenario 1 (Failure to perform implicit actions):
-
βΉοΈ Context: Fuel tank has 5 gallons of gas initially. The max capacity is 50 gallons.
+
βΉοΈ Context: Fuel tank has 5 gallons of gas initially. The max capacity is 50 gallons.
+
π§βπ» User Prompt: Fill the fuel tank until we are able to reach Rivermist. Oil costs
money so I just need to reach there, I don't need a full tank.
@@ -918,7 +1309,8 @@
Result & Error Analysis
π€ Model Text Response: It seems that the fuel tank is already full or nearly full, and I
+
π€ Model Text Response: It seems that the fuel tank is already full or nearly full,
+ and I
can't add more fuel beyond its capacity. Would you like me to display the current fuel status?
@@ -964,7 +1356,8 @@ Result & Error Analysis
Scenario 3 (LLMs incur unnecessary planning and thinking):
βΉοΈ Context: Twitter API has initially been authenticated.
-
π§βπ» User Prompt: I would appreciate it if you could share the sorted result on social
+
π§βπ» User Prompt: I would appreciate it if you could share the sorted result on
+ social
media, ensuring to tag 'current tech trend' and mention Julia, our insightful team.
@@ -982,7 +1375,8 @@
Result & Error Analysis
- π€ Model Text Response: It seems that I'm unable to authenticate with Twitter at the moment.
+
π€ Model Text Response: It seems that I'm unable to authenticate with Twitter at the
+ moment.
Please ensure your credentials are correct or try again later.
diff --git a/leaderboard.html b/leaderboard.html
index e29c8b49d..b78e26fb7 100644
--- a/leaderboard.html
+++ b/leaderboard.html
@@ -113,7 +113,7 @@ BFCL Leaderboard
@@ -137,19 +137,17 @@ BFCL Leaderboard
- FC = native support for function/tool calling.
+ FC = native support for function/tool calling.
+ Prompt = walk-around for function calling, using model's normal text generation capability.
Cost is calculated as an estimate of the cost per 1000 function calls, in USD.
Latency is measured in seconds.
- For Open-Source Models, the cost and latency are calculated when serving with vLLM using 8 V100 GPUs. The formula can be found in
- the blog.
- AST Summary is the unweighted average of the four test categories under AST Evaluation.
- Exec Summary is the unweighted average of the four test categories under Exec Evaluation.
Overall Accuracy is the unweighted average of all the sub-categories.
+ For details on score composition, please refer to our blog.
Click on column header to sort. If you would like to add
@@ -157,7 +155,7 @@
BFCL Leaderboard
href="https://discord.gg/grXXvj9Whz">discord.
- Models were evaluated using commit d7e52e5.
+ Models are evaluated using commit d7e52e5.
All the model response we obtained is available here.
To reproduce the results, please checkout our codebase at
this checkpoint.