Spark
Sign In
Prompt Chain

Generate and Review Code with AI Agents

Promptfoo example evaluating ElevenLabs voice agents on multi-turn conversation quality, tool use, cost.

Copy chain
Promptfoo
Own this? Claim it
73
Spark score
out of 100
Updated last month
Version code-scan-action-0.1
Models
universal
Add to Favorites

Why it matters

Leverage AI agents to automate code generation, review, and debugging processes, enhancing developer productivity and code quality.

Outcomes

What it gets done

01

Generate code snippets based on requirements.

02

Review generated code for errors and style adherence.

03

Assist in debugging identified code issues.

04

Orchestrate multiple AI agents for complex coding tasks.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-agents | bash

Steps

Steps in the chain

01
Setup ElevenLabs API key
02
Initialize example project
03
Run evaluation
04
View results in UI
05
Analyze conversation flow
06
Review evaluation scores
07
Assess tool usage
08
Evaluate response quality
09
Track cost metrics

Overview

Agents

A Promptfoo example evaluating ElevenLabs conversational voice agents across 8 weighted quality criteria (greeting, understanding, empathy, resolution) using simulated multi-turn conversations and tool-call tracking. Use when testing an ElevenLabs voice agent's conversation quality, context retention, or tool-calling behavior across multiple turns.

What it does

A Promptfoo example testing and evaluating ElevenLabs voice AI conversational agents across multi-turn dialogues, measuring conversation quality, tool usage, and cost and latency. It supports three conversation input formats: plain text treated as the first user message, multi-line text with User and Agent role prefixes, and structured JSON with an explicit array of speaker and message turns. Agent behavior is configured via an agentConfig block naming the agent, its system prompt, first message, language, ElevenLabs voiceId, backing llmModel, temperature, and maxTokens. Eight named evaluation-criteria presets each carry their own weight and pass threshold: greeting (weight 0.8, threshold 0.8), understanding (1.0, 0.9), accuracy (1.0, 0.9), helpfulness (0.9, 0.8), professionalism (0.7, 0.8), empathy (0.8, 0.7), efficiency (0.7, 0.7), and resolution (1.0, 0.8), each scored automatically on a 0-1 scale. A simulatedUser block drives automated conversation testing by giving the test harness its own persona prompt, temperature, and responseStyle of concise, verbose, casual, or formal, so the agent is tested against realistic, varied user behavior rather than fixed scripts. Seven example agent tools are named: get_weather, search_knowledge_base, create_ticket, send_email, get_order_status, schedule_callback, and transfer_agent, exercising the agent's tool-calling behavior as part of the evaluation.

When to use - and when NOT to

Use when testing an ElevenLabs conversational voice agent's dialogue quality, context retention across turns, tool-calling behavior, or cost and latency per conversation. Not the right example for single-turn text generation or non-conversational TTS or STT tasks - those are separate ElevenLabs provider examples in the same family.

Inputs and outputs

Input is an ELEVENLABS_API_KEY environment variable, an agentConfig defining the agent's persona and model, a conversation in one of the three supported formats, and optionally a simulatedUser persona for automated multi-turn testing. Output is per-criterion evaluation scores from 0 to 1 against the eight weighted presets, tool-call records, and per-conversation and per-turn cost tracking, viewable via the Promptfoo UI.

config:
  agentConfig:
    name: Customer Support Agent
    prompt: You are a helpful, empathetic customer support agent...
    firstMessage: Hi! I'm here to help. What can I do for you today?
    language: en
    voiceId: 21m00Tcm4TlvDq8ikWAM
    llmModel: gpt-4o
    temperature: 0.7
    maxTokens: 500

Integrations

Built on ElevenLabs's Conversational AI API via Promptfoo's ElevenLabs provider, with the agent's underlying LLM configurable, gpt-4o in the example, and its voice drawn from ElevenLabs's voice library; points to ElevenLabs's conversational AI and agent-configuration docs.

Who it's for

Developers building or QA-testing ElevenLabs voice agents who need automated, scored multi-turn conversation testing - greeting, understanding, empathy, resolution, tool use - rather than manually talking to the agent to check quality.

Source README

provider-elevenlabs/agents (ElevenLabs Conversational Agents)

You can run this example with:

npx promptfoo@latest init --example provider-elevenlabs/agents
cd provider-elevenlabs/agents

Test and evaluate ElevenLabs voice AI agents with multi-turn conversations.

What this tests

  • Agent conversation quality: Multi-turn dialogue handling
  • Evaluation criteria: Greeting, understanding, accuracy, helpfulness
  • Simulated user behavior: Automated conversation testing
  • Tool usage: Agent tool calls and responses
  • Cost and latency metrics

Setup

Set your ElevenLabs API key:

export ELEVENLABS_API_KEY=your_api_key_here

Run the example

npx promptfoo@latest eval -c ./promptfooconfig.yaml

Or view in the UI:

npx promptfoo@latest eval -c ./promptfooconfig.yaml
npx promptfoo@latest view

What to look for

  1. Conversation flow: How well the agent maintains context across turns
  2. Evaluation scores: Automated grading on multiple criteria (0-1 scale)
  3. Tool usage: When and how the agent calls available tools
  4. Response quality: Agent's ability to understand and respond accurately
  5. Cost tracking: Per-conversation and per-turn costs

Conversation formats

This example supports multiple input formats:

1. Plain text (treated as first user message)

prompts:
  - 'Hello, I need help with my order'

2. Multi-line with role prefixes

prompts:
  - |
    User: Hi, what's the weather like?
    Agent: I'd be happy to help! Where are you located?
    User: I'm in San Francisco

3. Structured JSON

prompts:
  - |
    {
      "turns": [
        {"speaker": "user", "message": "Hello"},
        {"speaker": "agent", "message": "Hi! How can I help?"},
        {"speaker": "user", "message": "I need support"}
      ]
    }

Agent configuration

Customize the agent behavior:

config:
  agentConfig:
    name: Customer Support Agent
    prompt: You are a helpful, empathetic customer support agent...
    firstMessage: Hi! I'm here to help. What can I do for you today?
    language: en
    voiceId: 21m00Tcm4TlvDq8ikWAM
    llmModel: gpt-4o
    temperature: 0.7
    maxTokens: 500

Evaluation criteria

Common criteria presets available:

  • greeting - Professional greeting (weight: 0.8, threshold: 0.8)
  • understanding - Accurate intent understanding (weight: 1.0, threshold: 0.9)
  • accuracy - Correct information (weight: 1.0, threshold: 0.9)
  • helpfulness - Helpful responses (weight: 0.9, threshold: 0.8)
  • professionalism - Professional tone (weight: 0.7, threshold: 0.8)
  • empathy - Empathetic responses (weight: 0.8, threshold: 0.7)
  • efficiency - Concise responses (weight: 0.7, threshold: 0.7)
  • resolution - Problem resolution (weight: 1.0, threshold: 0.8)

Simulated user

Configure the simulated user's behavior:

simulatedUser:
  prompt: Act as a customer who is frustrated but polite
  temperature: 0.8
  responseStyle: casual # concise | verbose | casual | formal

Available tools

Example tools for agents:

  • get_weather - Get current weather
  • search_knowledge_base - Search documentation
  • create_ticket - Create support ticket
  • send_email - Send email notification
  • get_order_status - Check order status
  • schedule_callback - Schedule callback
  • transfer_agent - Transfer to human agent

Learn more

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.