Evaluate Voice AI Conversations

This example configures promptfoo to evaluate xAI's Grok Voice Agent API for real-time voice AI conversations. It runs test suites against xAI's realtime WebSocket voice endpoint, using the grok-voice-think-fast-1.0 model selected via the ?model= query parameter on the websocket URL. The configuration supports xAI's five built-in voices - Ara (the default), Rex, Sal, Eve, and Leo - and can drive the agent through custom function tools that it calls mid-conversation, such as a get_weather or set_reminder function. Functions can be defined inline in the promptfoo config or loaded from an external YAML or JSON file, for example a tools.yaml referenced via file://tools.yaml. The Voice Agent API also ships three built-in tools - web_search, x_search, and file_search - that the model can call directly without any custom function definitions.

Use this example when you are building or testing a voice-based AI agent on xAI's Grok Voice Agent API and need a repeatable, automated way to check that it behaves correctly across conversation scenarios - for example asserting that a specific custom function was actually called during a test run. It is a good starting point for regression-testing voice-agent behavior before shipping changes to prompts, tools, or audio settings. It is not a general text-based LLM eval harness - the API and configuration are specific to xAI's realtime voice/audio product, and it assumes billed access to xAI's Voice Agent API at $0.05 per minute of audio, not a free evaluation path.

Input is a promptfoo YAML configuration: the provider block selects the model via the websocket ?model= parameter, sets one of the five available voices, and can attach input/output audio format settings - audio/pcm, audio/pcmu, or audio/pcma - at sample rates from 8000 Hz up to 48000 Hz. Tool and function definitions, whether inline or loaded from an external file, describe what the voice agent can call mid-conversation. Test cases carry vars (such as a spoken question) and assert blocks; a JavaScript assertion can inspect output.functionCalls to confirm a specific function name was invoked, for example checking that a "set the volume to 50%" prompt triggered a set_volume call. Output is promptfoo's standard eval report over these test cases, produced by running promptfoo's eval command.

The example integrates with xAI's Grok Voice Agent API directly, authenticating through the XAI_API_KEY environment variable:

npx promptfoo@latest init --example xai/voice
cd xai/voice
export XAI_API_KEY=your-api-key

For local testing, testing behind a proxy, or connecting to an endpoint that needs extra query parameters, the provider config accepts a custom websocketUrl in place of the default connection. Everything runs on top of the standard promptfoo CLI, so it fits into the same test-and-CI workflow as other promptfoo examples.

Teams building voice AI products on xAI's Grok Voice Agent API who need an automated way to evaluate conversation quality, verify function-calling behavior, and check audio configuration before shipping a release - rather than manually talking to the voice agent to check each change by hand.