Evaluate Strands Agents with Promptfoo

This example demonstrates how to evaluate the Strands Agents SDK with promptfoo. Strands Agents is an open-source, model-driven AI agent framework developed by AWS. The example builds a Strands agent with two custom tools defined via the @tool decorator - get_weather (returns mock weather data for New York, London, Tokyo, Paris, Seattle, and San Francisco) and convert_temperature (converts between Fahrenheit and Celsius) - and exercises it through promptfoo's assertion system.

Use this example when your agent is built on the Strands Agents SDK and you want to verify its tool usage, multi-tool chaining, and graceful handling of edge cases (like unknown cities) through promptfoo assertions. It is not a general Strands tutorial; it documents specifically how to wire a Strands agent into promptfoo's Python provider and test it with a representative assertion mix.

Requires Python 3.9+ and an OpenAI API key by default (or another supported provider). Install dependencies with pip install -r requirements.txt, which pulls in strands-agents[openai] (the Strands SDK with OpenAI support) and pydantic (data validation required by Strands), then set OPENAI_API_KEY. To use Anthropic instead, install strands-agents[anthropic], set ANTHROPIC_API_KEY, and modify agent.py to use AnthropicModel instead of OpenAIModel; Amazon Bedrock is supported via strands-agents[bedrock]. Scaffold and run:

npx promptfoo@latest init --example integration-strands-agents
cd integration-strands-agents

Then npx promptfoo eval to run the evaluation and npx promptfoo view to see results in the web UI. The agent lives in agent.py, and agent_provider.py exposes a call_api function that promptfoo's Python provider calls to interact with it. The config includes 5 test cases: a basic weather query for New York (contains-any, llm-rubric, latency), a London query verifying temperature format (contains-any, javascript, latency), a case-insensitive Tokyo query (icontains, javascript, latency), a multi-tool weather-plus-conversion chain (llm-rubric, javascript, latency), and an unknown-city fallback test (icontains, not-contains, latency).

Integrates the Strands Agents SDK (AWS's model-driven agent framework, with OpenAI, Anthropic, or Amazon Bedrock as the underlying model provider) with promptfoo's Python provider interface. Assertion types used: latency (applied to every test via defaultTest, enforcing a 30-second response ceiling), contains-any (verifies expected city names/weather data from the mock tool appear), icontains (case-insensitive city-name matching), not-contains (confirms unknown cities are handled without error messages), javascript (validates temperature-symbol formatting and response length), and llm-rubric (semantically checks that the agent correctly chains weather lookup with temperature conversion).

Teams building agents on the Strands Agents SDK who want a representative promptfoo assertion suite - tool-usage checks, multi-tool chaining, graceful fallback, and rubric-graded semantic correctness - as a starting template for their own agent evals.