Test AWS Bedrock Agents with Promptfoo

amazon-bedrock/agents (AWS Bedrock Agents Example)

This example demonstrates how to use AWS Bedrock Agents with promptfoo to test and evaluate deployed AI agents, including both single-agent and multi-agent scenarios.

You can run this example with:

npx promptfoo@latest init --example amazon-bedrock/agents
cd amazon-bedrock/agents

Prerequisites

1. An AWS account with Bedrock Agents access

2. One or more deployed Bedrock agents (get agent IDs from the AWS Console)

3. AWS credentials configured (via environment variables, AWS CLI, or IAM role)

4. Install the required AWS SDK:

   npm install @aws-sdk/client-bedrock-agent-runtime
   

Setup

1. Get your Agent ID(s):

   • Go to the AWS Bedrock Console
   • Navigate to Agents
   • Copy your agent ID(s) (format: ABCDEFGHIJ)

2. Configure AWS Credentials (choose one method):

   Via environment variables:

   export AWS_ACCESS_KEY_ID=your_access_key
   export AWS_SECRET_ACCESS_KEY=your_secret_key
   export AWS_REGION=us-east-1
   

   Via AWS CLI profile:

   aws configure --profile my-bedrock-profile
   

   Via IAM role (if running on EC2/Lambda)

3. Choose your configuration:

   • promptfooconfig.yaml: Basic single-agent configuration
   • promptfooconfig.multi-agent.yaml: Advanced multi-agent system configuration

Running the Examples

Single Agent Example

# Run basic agent evaluation
npx promptfoo eval -c promptfooconfig.yaml

# View results in the web UI
npx promptfoo view

Multi-Agent Example

# Run multi-agent system evaluation
npx promptfoo eval -c promptfooconfig.multi-agent.yaml

# View results in the web UI
npx promptfoo view

Configuration Options

Basic Usage

providers:
  - bedrock-agent:YOUR_AGENT_ID

Advanced Single Agent Configuration

providers:
  - id: bedrock-agent:my-agent
    config:
      agentId: YOUR_AGENT_ID
      agentAliasId: PROD_ALIAS # Optional: specific version/alias
      region: us-east-1 # AWS region
      sessionId: session-123 # Maintain conversation state
      enableTrace: true # Get detailed execution traces
      memoryId: SHORT_TERM_MEMORY # or LONG_TERM_MEMORY

Multi-Agent System Configuration

providers:
  # Technical Support Agent
  - id: tech-agent
    provider: bedrock-agent:TECH_AGENT_ID
    config:
      agentId: TECH_AGENT_ID
      agentAliasId: TECH_ALIAS_ID
      region: us-east-1
      enableTrace: true
      memoryId: LONG_TERM_MEMORY

  # Billing Agent
  - id: billing-agent
    provider: bedrock-agent:BILLING_AGENT_ID
    config:
      agentId: BILLING_AGENT_ID
      agentAliasId: BILLING_ALIAS_ID
      region: us-east-1
      enableTrace: true

Features

Session Management

The provider supports maintaining conversation state across multiple interactions:

config:
  sessionId: my-session-123 # Use the same session ID for related queries

Memory Integration

Enable agent memory for context-aware responses:

config:
  memoryId: LONG_TERM_MEMORY # or SHORT_TERM_MEMORY

Trace Information

Get detailed execution traces including tool calls and reasoning:

config:
  enableTrace: true # Response will include trace metadata

Testing Scenarios

Single Agent Tests

The basic config includes tests for:

• Basic agent responses
• Tool/function calling (e.g., calculator)
• Memory retention
• Multi-turn conversations

Multi-Agent System Tests

The multi-agent config includes tests for:

• Specialized agent capabilities (technical, billing, product)
• Cross-functional issue handling
• Agent collaboration and coordination
• Escalation management
• Performance and latency validation

Multi-Agent System Architecture

The multi-agent example demonstrates a customer support system with specialized agents:

Customer Query
     ↓
[Supervisor Agent] ← Monitors & Routes
     ↓
┌─────────────────┬─────────────────┬──────────────────┐
│  Tech Agent     │  Billing Agent  │  Product Agent   │
│  (Technical)    │  (Payments)     │  (Recommendations)│
└─────────────────┴─────────────────┴──────────────────┘

Troubleshooting

1. Authentication Error: Ensure AWS credentials are properly configured
2. Agent Not Found: Verify the agent ID and region
3. Permissions Error: Check IAM permissions for bedrock:InvokeAgent
4. Timeout: Large agent responses may take time; adjust timeout if needed
5. Multi-Agent Issues: Ensure all agent IDs and aliases are correct in the config

IAM Permissions

Your AWS credentials need the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["bedrock:InvokeAgent"],
      "Resource": "arn:aws:bedrock:*:*:agent/*"
    }
  ]
}

Learn More

• AWS Bedrock Agents Documentation
• Promptfoo Documentation
• Bedrock Agents Samples