Test AWS Bedrock Agents with Promptfoo
Promptfoo eval config for testing AWS Bedrock Agents, including conversation memory retention across a session.
Why it matters
Evaluate and test deployed AI agents on AWS Bedrock, covering both single and multi-agent configurations.
Outcomes
What it gets done
Test single-agent AWS Bedrock deployments.
Evaluate multi-agent AWS Bedrock scenarios.
Automate AI agent testing and validation.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-bedrock-agents | bash Overview
Bedrock Agents
This Promptfoo example tests AWS Bedrock Agents via the bedrock-agent provider, covering a capability question, a calculation, and a session-memory-retention pair that checks whether the agent recalls information given in an earlier turn. Use it as a template when testing an AWS Bedrock Agent's responses and, specifically, whether it retains information across turns in a session.
What it does
This Promptfoo config tests AWS Bedrock Agents via the bedrock-agent:<id> provider syntax. Two provider setups are shown: a basic one (agentId, agentAliasId, region) and an advanced one that adds sessionId to maintain conversation state, enableTrace for detailed trace information, memoryId (SHORT_TERM_MEMORY or LONG_TERM_MEMORY), and optional AWS credentials (accessKeyId, secretAccessKey, sessionToken, SSO profile) that default to AWS's standard credential chain when not set.
The test suite covers three behaviors: a capability question ("Tell me about your capabilities" asserted to contain "agent"), a calculation ("what is 15 multiplied by 27" / a sum-of-100-and-250 test asserted to contain "350" and be non-empty), and a memory-retention pair - telling the agent "my favorite color is blue" in one turn, then asking "what is my favorite color?" in a later turn within the same session, asserted to contain "blue" - which specifically exercises the agent's session memory rather than just its single-turn responses.
When to use - and when NOT to
Use it as a template for testing an AWS Bedrock Agent's responses, including whether it retains information across turns in a session. Do not use it if you're not using AWS Bedrock Agents, or if your evaluation doesn't need memory-retention testing - the basic single-turn provider config covers that case on its own.
Inputs and outputs
Input: the YAML config (prompts, per-provider agent ID/alias/region, and optional session/trace/memory/credential settings) plus environment variables (AWS_BEDROCK_REGION, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN). Output: Promptfoo's evaluation report, scoring each response against its contains / not-empty assertions.
Integrations
Uses Promptfoo's bedrock-agent provider to call AWS Bedrock Agents directly, authenticating via AWS's default credential chain, explicit access keys, or an AWS SSO profile.
Who it's for
Teams building on AWS Bedrock Agents who want to test agent responses and session-based memory retention as part of a Promptfoo eval suite.
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.