Langgraph Agent - Output dataframe from a tool and use it in another

[hepbc]

Hi guys: I am building a simple agent that takes input and executes tools till it gets an END msg. I have bound a bunch of tools to the agent. One of the tools (get_price_history) outputs a pandas dataframe (dates and stock prices on those dates) and another (plot_chart) plots dates and prices. The expected output is that the get_price_history tool gives the prices and the agent routes this data to the plot_chart tool for the chart.

@tool
def get_price_history(ticker: str, start_date: str, end_date: str) -> pd.DataFrame:

@tool
def plot_chart(x_axis: list, y_axis: list):

This works ok if the price data is upto a few days. Otherwise, it takes the first few days and the last few days and does not output the graph correctly: see example of a 6 month chart below.

DatetimeIndex: 125 entries, 2024-01-29 to 2024-07-26
Series name: Adj Close
Non-Null Count  Dtype  
--------------  -----  
125 non-null    float64
dtypes: float64(1)
memory usage: 2.0 KB
None
Back to the model!

Calling: {'name': 'plot_chart', 'args': {'x_axis': ['2024-01-29', '2024-01-30', '2024-01-31', '2024-02-01', '2024-02-02', '2024-07-22', '2024-07-23', '2024-07-24', '2024-07-25', '2024-07-26'], 'y_axis': [408.227386, 407.101501, 396.131622, 402.309052, 409.721954, 442.940002, 444.850006, 428.899994, 418.399994, 425.269989]}, 'id': 'call_KsGzivmtBPOFuWBPmTSQ31kZ', 'type': 'tool_call'}

I noticed that I am converting the tool output to a string in the take_action fn of the agent:

results.append(ToolMessage(tool_call_id=t['id'], name=t['name'], content=str(result)))

This seems to be losing data as most of it is replaced by "..."

I tried passing as json but get a number of tokens limit error.

Can someone pls guide me here? Is it possible to pass a dataframe as is - if not how can I maintain the integrity of the string output so all the price data is preserved? More generally how can I pass output in different formats between tools.

Look forward to the community working its magic.
Thanks
BC

[gbaian10]

If you use ToolNode, it will return a ToolMessage, and the content of the ToolMessage will be converted to a str or a list.

Here is my current approach. If I want a tool result (type is Any), I will use artifacts, allowing my tool func to return a tuple of (content, artifact).

My content might be an empty string because I am not handing it over to the LLM to answer questions.

I receive the results of artifacts at subsequent nodes and process it.

[hepbc]

Thanks @gbaian10 ! Will look this up.
However, how do I pass the artifact to another tool - in this case, the prices df to the plot tool? The plot tool is part of the agent run (the query to plot a chart) and hence it's called by the agent itself.
Thanks again.

[gbaian10]

@hepbc Are you using create_react_agent to build the graph?

from langgraph.prebuilt import create_react_agent

I usually build the graph myself.

[hepbc]

@gbaian10 Have built the graph myself (code below). Could you suggest a few tweaks?!
Am novice level and hence all these questions, sorry!

class Agent:

    def __init__(self, model, tools, system=""):
        self.system = system
        graph = StateGraph(AgentState)
        graph.add_node("llm", self.call_openai)
        graph.add_node("action", self.take_action)
        graph.add_conditional_edges(
            "llm",
            self.exists_action,
            {True: "action", False: END}
        )
        graph.add_edge("action", "llm")
        graph.set_entry_point("llm")
        self.graph = graph.compile()
        self.tools = {t.name: t for t in tools}
        self.model = model.bind_tools(tools)

    def exists_action(self, state: AgentState):
        result = state['messages'][-1]
        return len(result.tool_calls) > 0

    def call_openai(self, state: AgentState):
        messages = state['messages']
        if self.system:
            messages = [SystemMessage(content=self.system)] + messages
        message = self.model.invoke(messages)
        return {'messages': [message]}

    def take_action(self, state: AgentState): 
        tool_calls = state['messages'][-1].tool_calls
        results = []
        for t in tool_calls:
            print(f"Calling: {t}")
            if not t['name'] in self.tools:      # check for bad tool name from LLM
                print("\n ....bad tool name....")
                result = "bad tool name, retry"  # instruct LLM to retry if bad
            else:
                result = self.tools[t['name']].invoke(t['args'])
            #parsed_result = tools_parser(result, config)
            results.append(ToolMessage(tool_call_id=t['id'], name=t['name'], content=str(result)))
        print("Back to the model!")
        return {'messages': results}

[gbaian10]

langchain-ai/langchain#25095 (comment)

Currently, for some reasons, the DataFrame seems unable to be passed in artifacts.
Waiting for a fix.🥺

[gbaian10]

update langchain-core to 0.2.38

I'm not sure what get_price_history and plot_chart do, so I just created them randomly.

import matplotlib

matplotlib.use("Agg")
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from dotenv import load_dotenv
from langchain_core.messages import HumanMessage, SystemMessage, ToolMessage
from langchain_core.tools import BaseTool, tool
from langchain_openai import ChatOpenAI
from langgraph.graph import END, START, MessagesState, StateGraph
from langgraph.prebuilt import ToolNode, tools_condition

load_dotenv()


class Agent:
    def __init__(
        self, model: ChatOpenAI, tools: list[BaseTool], system: str = ""
    ) -> None:
        self.system = system
        self.model = model.bind_tools(tools)
        graph = StateGraph(MessagesState)
        graph.add_node("llm", self.call_openai)
        graph.add_node("pandas", ToolNode(tools=tools))
        graph.add_node("plot", self.plot_chart)

        graph.add_edge(START, "llm")
        graph.add_conditional_edges(
            "llm", tools_condition, {"tools": "pandas", END: END}
        )
        graph.add_edge("pandas", "plot")
        graph.add_edge("plot", END)

        self.graph = graph.compile()

    def call_openai(self, state: MessagesState) -> MessagesState:
        messages = state["messages"]
        if self.system:
            messages = [SystemMessage(content=self.system)] + messages
        return {"messages": [self.model.invoke(messages)]}

    def plot_chart(self, state: MessagesState) -> None:
        tool_message: ToolMessage = state["messages"][-1]
        table = tool_message.artifact
        x_axis, y_axis = table["Date"], table["Price"]

        plt.figure(figsize=(10, 5))
        plt.plot(x_axis, y_axis, marker="o")
        plt.title("Price History")
        plt.xlabel("Date")
        plt.ylabel("Price")
        plt.grid(True)
        plt.xticks(rotation=45)
        plt.tight_layout()
        # plt.show()

        path = "123.png"
        plt.savefig(path)
        plt.close()
        print(f"Your chart is saved as {path}")


@tool(response_format="content_and_artifact")
def get_price_history(start_date: str, end_date: str) -> tuple[str, pd.DataFrame]:
    """Get price history data."""
    dates = pd.date_range(start=start_date, end=end_date)
    prices = np.random.rand(len(dates)) * 100
    return "", pd.DataFrame({"Date": dates, "Price": prices})


model = ChatOpenAI(model="gpt-4o-2024-08-06")
tools = [get_price_history]

graph = Agent(model, tools).graph
graph.invoke(
    {
        "messages": [
            HumanMessage(
                content="I want to see the price history between 2023-01-01 and 2023-01-10"
            )
        ]
    }
)

[hepbc]

Thanks for all your help @gbaian10 !
Am working on Google Colab so the latest versions are installed:

langchain==0.2.16
langchain-core==0.2.38

However, looks like the update is still not 100% stable:

1. The first time I ran your block, it worked partly, i.e. the image file '123.png' is created, but after that I get the following error:

Your chart is saved as 123.png
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/IPython/core/formatters.py](https://localhost:8080/#) in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--> 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

10 frames
[/usr/local/lib/python3.10/dist-packages/pandas/core/generic.py](https://localhost:8080/#) in __nonzero__(self)
   1517     @final
   1518     def __nonzero__(self) -> NoReturn:
-> 1519         raise ValueError(
   1520             f"The truth value of a {type(self).__name__} is ambiguous. "
   1521             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

2. The second time, I made the following edits and it gave the ValueError:

graph = Agent(model, tools).graph
res = graph.invoke(
    {
        "messages": [
            HumanMessage(
                content="I want to see the price history between 2023-01-01 and 2023-01-10"
            )
        ]
    }
)
Your chart is saved as 123.png

for msg in res["messages"]:
  print(msg)

content='I want to see the price history between 2023-01-01 and 2023-01-10' id='a797572c-98ce-4a23-b26e-08cb50541990'
content='' additional_kwargs={'tool_calls': [{'id': 'call_dJIGVuDrAiaOGb0TcBxEuNUk', 'function': {'arguments': '{"start_date":"2023-01-01","end_date":"2023-01-10"}', 'name': 'get_price_history'}, 'type': 'function'}], 'refusal': None} response_metadata={'token_usage': {'completion_tokens': 31, 'prompt_tokens': 72, 'total_tokens': 103}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_9e15ccd6a4', 'finish_reason': 'tool_calls', 'logprobs': None} id='run-f721e8e0-9310-4bd5-8490-4a2d3598c477-0' tool_calls=[{'name': 'get_price_history', 'args': {'start_date': '2023-01-01', 'end_date': '2023-01-10'}, 'id': 'call_dJIGVuDrAiaOGb0TcBxEuNUk', 'type': 'tool_call'}] usage_metadata={'input_tokens': 72, 'output_tokens': 31, 'total_tokens': 103}
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
[<ipython-input-19-4ae9641a1287>](https://localhost:8080/#) in <cell line: 1>()
      1 for msg in res["messages"]:
----> 2   print(msg)

4 frames
[/usr/local/lib/python3.10/dist-packages/pandas/core/generic.py](https://localhost:8080/#) in __nonzero__(self)
   1517     @final
   1518     def __nonzero__(self) -> NoReturn:
-> 1519         raise ValueError(
   1520             f"The truth value of a {type(self).__name__} is ambiguous. "
   1521             "Use a.empty, a.bool(), a.item(), a.any() or a.all()."

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

Looks like it's blowing up on the 'messages' attr. Thanks again.

[gbaian10]

@hepbc
This seems to be an issue with the Expression output, which is likely another bug.
You can temporarily avoid this problem by assigning the last Expression to a variable.

Original

inputs = {"messages": [HumanMessage(content="I want to see the price history between 2023-01-01 and 2023-01-10")]}
graph.invoke(inputs)

Revised

inputs = {"messages": [HumanMessage(content="I want to see the price history between 2023-01-01 and 2023-01-10")]}
outputs = graph.invoke(inputs)

I will report this issue to Langchain again.