Evaluate Pydantic AI Agents with Promptfoo

This example demonstrates how to evaluate PydanticAI agents using promptfoo. PydanticAI is a Python agent framework that provides structured outputs and type safety for AI applications. The example builds a simple PydanticAI weather agent with structured output and runs it through promptfoo's Python provider so its behavior can be evaluated like any other promptfoo target.

Use this example when your agent is built with PydanticAI and you want to validate its structured outputs and tool usage through promptfoo's assertion system - JSON schema validation, JavaScript checks, Python checks, and LLM-rubric evaluations. It is not a general PydanticAI tutorial; it only shows the minimal wiring needed to run a PydanticAI agent through promptfoo's Python provider.

npx promptfoo@latest init --example integration-pydantic-ai
cd integration-pydantic-ai
pip install -r requirements.txt
export OPENAI_API_KEY=your_openai_api_key_here
npx promptfoo@latest eval
npx promptfoo@latest view

The example is structured around four files: agent.py (the PydanticAI weather agent with structured output), provider.py (the promptfoo Python provider that runs the agent), promptfooconfig.yaml (the evaluation configuration with diverse assertion types), and requirements.txt (the Python dependencies). It demonstrates JSON schema validation via is-json assertions alongside JavaScript, Python, and LLM-rubric assertion types, and shows how to evaluate the agent's tool usage as part of the eval.

Integrates PydanticAI (a Python agent framework for structured, type-safe AI outputs) with promptfoo's Python provider interface, requiring an OPENAI_API_KEY since the agent calls OpenAI.

Teams building agents with PydanticAI who want to evaluate structured output correctness, JSON schema compliance, and tool usage using promptfoo's assertion system instead of writing bespoke test harnesses.