Evaluate AI Agent Performance
An example evaluating CrewAI multi-agent performance with Promptfoo, including a note on real-LLM output reliability.
Why it matters
Leverage CrewAI agents and promptfoo to rigorously evaluate the performance and reliability of your AI agents. Ensure your agents meet desired quality standards before deployment.
Outcomes
What it gets done
Set up and run evaluations for CrewAI agent performance.
Integrate promptfoo for systematic AI agent testing.
Analyze agent outputs to identify areas for improvement.
Automate the process of AI agent quality assurance.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-integration-crewai | bash Steps
Steps in the chain
Overview
Integration Crewai
An example evaluating CrewAI multi-agent performance with Promptfoo, via a custom provider interface bridging CrewAI's orchestration framework to Promptfoo's evaluation harness. Use it when building CrewAI-based agents and evaluating their real-world reliability. Requires Python 3.10+, a compatible Node.js version, and an OpenAI API key.
What it does
An example showing how to evaluate CrewAI agent performance with Promptfoo. CrewAI is a framework for orchestrating role-playing, autonomous AI agents that collaborate to tackle complex tasks together.
When to use - and when NOT to
Use it when you're building CrewAI-based agents and want to evaluate their real-world performance with Promptfoo. It requires Python 3.10+, Node.js ^20.20.0 or >=22.22.0 (Node.js 20 support ends 2026-07-30, Node.js 24 LTS is recommended), and a valid OpenAI API key. The example's own reliability note is worth heeding: agent output from a real LLM isn't always reliable, especially on complex queries - it may fail to return valid JSON or return no response at all.
Inputs and outputs
Set the OpenAI API key via environment variable (recommended) or a .env file - if using the latter, uncomment python-dotenv in requirements.txt and reinstall. Install Python dependencies with pip install -r requirements.txt (openai, langchain, pydantic, and others install automatically as CrewAI's own dependencies), then install the Promptfoo CLI globally with npm install -g promptfoo. Run with:
npx promptfoo@latest init --example integration-crewai
cd integration-crewai
promptfoo eval
promptfoo view
Two files drive the example: agent.py (the CrewAI agent setup and Promptfoo provider interface) and promptfooconfig.yaml (prompts, providers, and tests). If you hit authentication errors, check the OpenAI API key is set correctly, valid, has sufficient quota, and is accessible to the Python process.
Integrations
Bridges CrewAI's multi-agent orchestration framework with Promptfoo's evaluation harness via a custom provider interface in agent.py; depends on OpenAI's API as the underlying LLM. The setup spans two runtimes: the CrewAI agent itself runs in Python, while Promptfoo's CLI (which drives and scores the evaluation) runs in Node.js.
Who it's for
Developers building CrewAI multi-agent systems who need to systematically evaluate agent output quality and reliability rather than spot-checking runs manually.
Source README
integration-crewai (CrewAI Integration)
This example shows how to use CrewAI agents with promptfoo to evaluate AI agent performance.
What is CrewAI?
CrewAI is a framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
Quick Start
You can run this example with:
npx promptfoo@latest init --example integration-crewai
cd integration-crewai
Prerequisites
This example requires the following:
- Python 3.10+
- Node.js ^20.20.0 or >=22.22.0 (Node.js 20 support ends July 30, 2026; Node.js 24 LTS recommended)
- OpenAI API Key - You MUST have a valid OpenAI API key to run this example
Environment Setup
You need to set the OpenAI API key. Choose one of these methods:
Option 1: Environment Variable (Recommended)
export OPENAI_API_KEY=your-api-key-here
Option 2: .env File
Create a .env file in this directory:
OPENAI_API_KEY=your-api-key-here
If using a .env file, uncomment python-dotenv in requirements.txt and reinstall dependencies.
Installation
Install Python packages:
pip install -r requirements.txt
Note: The openai package and other dependencies (langchain, pydantic, etc.) will be automatically installed as dependencies of crewai.
Install promptfoo CLI:
npm install -g promptfoo
Files
agent.py: Contains the CrewAI agent setup and promptfoo provider interfacepromptfooconfig.yaml: Configures prompts, providers, and tests for evaluation
Note on Reliability
When using a real LLM, you may notice that the agent's output is not always reliable, especially for more complex queries. For example, the agent may fail to return valid JSON or may not return a response at all. This is a common challenge when working with LLMs.
Running the Evaluation
Run the evaluation:
promptfoo eval
Explore results in browser:
promptfoo view
Troubleshooting
If you see authentication errors:
- Ensure your OpenAI API key is set correctly
- Verify the key is valid and has sufficient quota
- Check that the environment variable is accessible to the Python process
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.