Test OpenAI Tool Calls

This promptfoo example evaluates OpenAI's tools-calling capability through the Chat Completions API. It defines a custom weather-lookup tool and tests whether the model correctly calls that function when asked about weather, passes the right location parameter for the city mentioned, handles a variety of cities including international ones, and formats its responses consistently.

Use this example when you need to validate that a model reliably calls the right tool with the right arguments across varied inputs, rather than just checking the final text response. It's scoped to the Chat Completions tools API specifically - if you need to validate output transformation and JavaScript-based assertion logic more broadly, the same techniques shown here apply, but the example itself is built around one weather tool.

npx promptfoo@latest init --example openai-tools-call
cd openai-tools-call

npx promptfoo eval
npx promptfoo view

The is-valid-openai-tools-call assertion validates that the returned tool call is structurally correct, while custom JavaScript assertions can isolate and check specific parts of the response, such as the parsed location argument, for more detailed validation than a structural check alone provides.

Requires OPENAI_API_KEY, set via a .env file or the environment. Full detail on tool definitions and provider configuration is in OpenAI's function-calling guide and promptfoo's OpenAI provider documentation for tools and functions.

Developers building tool-using OpenAI agents who need a working reference for asserting on tool-call structure and argument correctness across a range of inputs before trusting the behavior in production.