OpenAI Tool Runner (experimental)

This package is a wrapper around the OpenAI API, allowing you to replace the baseURL to use it with Ollama and compatible models. It enables running tools in sequence without generating a response after each set of tool calls, unlike the standard "Input -> Tool(s) -> Response" flow described in OpenAI documentation. Instead, it supports "Input -> Tool(s) -> Tool(s) -> ... -> Response."

• Completer: Runs a completion with forced tool calls.
• Free Runner: Executes tools chosen by the LLM until a specific tool is used.
• Straight Runner: Enforces the order of tool calls in the provided toolchain.

These runners return only tool calls, useful for recursively continuing the completion process. For example, you can provide a tool like "provide_final_answer" for display in the frontend. This relies on prompt engineering to generate effective tool call flows.

Primarily tested with GPT-4o.

Use this only for experimentation. There is no error handling or other features to make it production-ready. Since it's a small amount of code, you can easily copy and paste it into your project to build upon.

Installation

npm install openai-tool-runner

export OPENAI_TOOL_RUNNER_DEFAULT_MODEL="gpt-4-turbo"
# gpt-4o otherwise, overwrite in createCompleter/Runner

Usage

createCompleter

import { createCompleter } from 'openai-tool-runner'

const completer = createCompleter({ apiKey: '...' })
const response = await completer({ messages, toolChain })

createFreeRunner

import { createFreeRunner, ToolChain } from 'openai-tool-runner'

const toolChain = new ToolChain({
  tools: [
    planResearchTool,
    webSearchTool,
    provideFinalAnswerTool,
    askUserTool,
  ],
  stopWhen: [
    provideFinalAnswerTool,
    askUserTool,
  ]
})

const runner = createFreeRunner({ systemMessage, chatHistory, toolChain })

for await (const message of runner()) {
  console.info(message)
}

createStraightRunner

import { createStraightRunner, ToolChain } from 'openai-tool-runner'

const toolChain = new ToolChain({
  tools: [
    searchTool,
    analyzeTool,
    provideFinalAnswerTool,
  ],
})

const runner = createStraightRunner({ systemMessage, chatHistory, toolChain })

for await (const message of runner()) {
  console.info(message)
}

create...Message

import { createSystemMessage, createUserMessage } from 'openai-tool-runner'

const systemMessage = createSystemMessage(`You are...`)
const userMessage = createUserMessage(`What is...`)

ToolChain

const provideFinalAnswerTool = new ProvideFinalAnswerTool()
const webSearchTool = new WebSearchTool(tavilyApiKey)

const toolChain = new ToolChain({
  tools: [
    webSearchTool,
    provideFinalAnswerTool,
  ],
  stopWhen: [
    provideFinalAnswerTool,
  ]
})

const response = await completer({ messages, toolChain })

Define a Tool

I chose not to use LangChain, but this isn't very different—just less sophisticated. You can still wrap a LangChain tool in it.

My initial idea was to make tool responses multi-step using generator functions, unlike LangChain. I had it working initially, but removed it during refactoring. This also involves some frontend considerations.

import type { ToolInterface } from 'openai-tool-runner'

export class WebSearchTool implements ToolInterface {
  name = 'web_search'
  description = 'Use this tool whenever you need to find information online to become more confident in your answers. Especially good for local information and recent events. You can use this tool mutliple times simultaneously, each call with multiple queries.'
  inputSchema = z.object({
    queries: z.array(z.string()).describe('The queries to search for.'),
  })

  outputSchema = z.object({
    results: z.array(z.any()).describe('The search results.'),
  })

  constructor(private tavilyApiKey: string) {}

  async run(args: z.infer<typeof this.inputSchema>): Promise<z.infer<typeof this.outputSchema>> {
    const { queries } = args

    const searchTool = new TavilySearchResults({
      maxResults: 1,
      apiKey: this.tavilyApiKey,
    })

    const results: string[] = []

    for (const query of queries) {
      results.push(JSON.parse(await searchTool.invoke(query)))
    }

    return { results }
  }
}

Nested Agents

Using a free runner as the main chatbot, with additional free and straight runners as tools, could yield interesting results.

import { type AgentMessage, type ToolInterface, createCompleter, createRunner, createSystemMessage, createUserMessage } from 'openai-tool-runner'
import { PlanResearchTool, ProvideFinalAnswerTool, WebSearchTool } from './your-tools'

export class ResearchAgentTool implements ToolInterface {
  name = 'research_agent'
  description = 'Use this tool whenever you need to research something online. Especially useful for...'
  inputSchema = z.object({
    prompt: z.string().describe('The prompt for the research agent.'),
  })

  outputSchema = z.object({
    result: z.string().describe('The final result of the research.'),
  })

  constructor(
    private tavilyApiKey: string,
    private chatHistory: AgentMessage[] = [],
  ) {}

  async run(args: z.infer<typeof this.inputSchema>): Promise<z.infer<typeof this.outputSchema>> {
    const systemMessage = createSystemMessage(`You are a research agent. You...`)
    const planResearchTool = new PlanResearchTool()
    const provideFinalAnswerTool = new ProvideFinalAnswerTool()
    const webSearchTool = new WebSearchTool(tavilyApiKey)
    const toolChain = new ToolChain({
      tools: [
        planResearchTool,
        webSearchTool,
        provideFinalAnswerTool,
      ],
      stopWhen: [
        provideFinalAnswerTool,
      ]
    })

    const chatHistory = [...this.chatHistory, createUserMessage(args.prompt)]
    const runner = createRunner({ systemMessage, chatHistory, toolChain })
    const agentMessages: AgentMessage[] = []

    for await (const message of runner()) {
      agentMessages.push(message)
    }

    const result = agentMessages[agentMessages.length - 1].content

    return { result }
  }
}

Streaming

Instead of streaming individual tokens, you can stream entire messages. As models become faster, token streaming may become irrelevant and primarily a frontend animation. Streaming actions that take significant time, like API calls, is more important. The runners are async generators, meaning they yield messages (including tool calls and responses) as they arrive, rather than returning a single result.

I use Nuxt 3 for my frontend, which utilizes h3 and its handy function sendIterable. You can pass the runner to it. For Next.js or other frameworks, there should be similar solutions. Here's a part of an endpoint in my application:

export default defineEventHandler(async (event) => {
  const { chatHistory }: { chatHistory: AgentMessage[] } = await readBody(event)
  const { openaiApiKey, tavilyApiKey } = useRuntimeConfig(event)

  const systemMessage = createSystemMessage(`You are ...

  Today's date: ${new Date().toISOString().slice(0, 16)}
  Your knowledge cutoff: 2023-10`)

  const webSearchTool = new WebSearchTool(tavilyApiKey)
  const askWebsiteTool = new AskWebsiteTool(openaiApiKey)
  const provideFinalAnswerTool = new ProvideFinalAnswerTool()

  const toolChain = new ToolChain({
    tools: [
      webSearchTool,
      askWebsiteTool,
      provideFinalAnswerTool,
    ],
    stopWhen: [
      provideFinalAnswerTool,
    ]
  })

  return sendIterable(event, createFreeRunner({
    apiKey: openaiApiKey,
    systemMessage,
    chatHistory,
    toolChain
  }))
})

You can read the messages using the provided readToolStream function:

import { readToolStream } from 'openai-tool-runner'

let loading = true
const chatHistory = []

const stream = await fetch('/api/agent', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ chatHistory }),
}).then((res) => res.body.getReader())

readToolStream(stream, (message) => {
  chatHistory.push(message)
}, () => loading = false)