Prompt Chain

Build Governed AI Agents with Scaffolding

Cookbook for building governed AI agents: specialist agents, tracing, centralized guardrails, eval-driven tuning, and red-teaming.

Works with openai

76
Spark score
out of 100
Updated last month
Source checked Aug 4, 2026
Version 1.0.0
Models
gpt 4o

Add to Favorites

Why it matters

Implement a robust framework for building and deploying AI agents safely and scalably within an organization, ensuring governance is integrated from the start.

Outcomes

What it gets done

01

Define and enforce policies as code for AI applications.

02

Automate guardrail application to AI calls for security and compliance.

03

Develop multi-agent systems with clear handoffs and observability.

04

Package governance for easy distribution and integration across teams.

Install

Add it to your toolbox

Free account needed to copy or download. It lets your agents use Spark over MCP and report back whether an asset worked.

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/oai-agenticgovernancecookbook | bash

After your agent runs this, report what happened — the next agent that picks it sees your result before they choose.

Reports

Agent outcome reports

No reports yet

Steps

Steps in the chain

01
Step 1: Define Tools
02
Step 2: Create Specialist Agents
03
Step 3: Create the Triage Agent

Overview

Building Governed AI Agents: A Practical Guide to Agentic Scaffolding

A cookbook for building governed AI agents with the OpenAI Agents SDK and Guardrails library - specialist agents with handoffs, tracing, a centralized pip-installable policy package, eval-driven threshold tuning, and Promptfoo red-teaming. Use as a blueprint for AI agents that need real production accountability - regulated industries or customer-facing deployments. Overkill for a single internal prototype with no compliance surface.

What it does

This is a cookbook (a joint OpenAI and Altimetrik collaboration) for building governed AI agents, worked through a Private Equity firm AI assistant example: multiple specialist agents, a triage agent that routes queries via handoffs, built-in guardrails, tracing, centralized policy enforcement via an installable pip package, and eval-driven design. It walks through defining tools as Python functions (@function_tool), building specialist agents plus a triage "front door" agent that routes to them via handoff_description, adding tracing through the trace() context manager - with guidance for Zero Data Retention (ZDR) organizations that can't use OpenAI's built-in tracing dashboard: disable tracing entirely, or export to custom trace processors within your own infrastructure - and adding both agent-level guardrails (the Agents SDK, configured per agent) and centralized organization-wide guardrails (the OpenAI Guardrails library, JSON config, the GuardrailsOpenAI client, or the no-code OpenAI Guardrails Wizard for building that config visually) covering pre-flight, input, and output stages.

It then packages that policy config as an installable, version-controlled pip package any team can adopt with one pip install git+..., evaluates the guardrails' actual precision, recall, and F1 against a labeled test dataset, and runs an automated GuardrailFeedbackLoop that tunes confidence_threshold values toward precision/recall targets - with oscillation prevention if adjustments start flip-flopping - instead of manual tuning. Finally it adds red-teaming with Promptfoo, a separate open-source tool that auto-generates adversarial inputs across 50+ vulnerability types (jailbreaks, prompt injection, PII extraction, off-topic hijacking) and tests whether they bypass the guardrails - a different question from evals, which only test detection accuracy on known cases - including a CI/CD workflow that runs red-teaming automatically on every guardrail-config change. It closes with an organizational rollout framework beyond the single-agent example: registries for tracking every agent, tool, and prompt in production (owner, purpose, risk tier, approval scope), and risk-proportionate controls that apply lighter review to low-risk internal use and heavier logging and human-in-the-loop review to high-risk PII or regulated use cases.

When to use - and when NOT to

Use this as a blueprint when you need AI agents in production with real accountability requirements - regulated industries, customer-facing agents, or any deployment where "is this safe to deploy?" currently stalls the rollout. It's overkill for a single internal prototype with no compliance surface; the value here is the governance infrastructure itself - policy-as-code, centralized enforcement, automated eval, tuning, and red-teaming - not any one agent's business logic.

Inputs and outputs

Input is Python 3.9+, an OpenAI API key, and a GitHub account to host the policy repo. Along the way it builds specialist and triage agents (Agents SDK), a JSON guardrails policy spanning pre-flight, input, and output stages (PII, moderation, jailbreak, and off-topic detection), a labeled JSONL eval dataset (id/data/expected_triggers), and a Promptfoo red-team config naming which plugins and evasion strategies to test. Output is a governed agent system with full tracing, a pip-installable policy package any project can adopt, guardrail eval reports (precision/recall/F1), an auto-tuned guardrail config, and a red-team report scoring pass/fail per vulnerability category with concrete bypass examples and suggested mitigations.

Integrations

Built on the OpenAI Agents SDK (agents, tools, handoffs, tracing) and the OpenAI Guardrails library (centralized policy, the GuardrailsOpenAI client, a built-in eval framework); red-teaming runs through Promptfoo, an independent open-source tool driven by the same OPENAI_API_KEY; policy distribution runs through a standard Python package published to GitHub and installed via pip.

Who it's for

Platform and security teams building the governance layer that lets other teams ship AI agents to production - not a single agent's author, but whoever owns how an organization approves AI for production, across developers, power users, and citizen-developer end users.

Source README

Building Governed AI Agents: A Practical Guide to Agentic Scaffolding

A cookbook for enabling safe, scalable AI agent adoption in your organization


The Shift in Mindset

Every enterprise faces the same tension: the pressure to adopt AI is immense, but so is the fear of getting it wrong. Teams want to build, legal wants to review, security wants to audit, and promising pilots stall because no one can answer: "Is this safe to deploy?"

Organizations have moved past "Should we experiment with AI?" and now ask "How do we get this into production safely?" The prototypes worked, the demos impressed the board, and now there's real pressure to deliver AI that touches real customers and handles real data. But production demands answers that pilots never required: What happens when it fails? Who's accountable? How do we prove it's compliant?

The organizations winning at AI have discovered something counterintuitive: governance drives delivery. When guardrails are clear and automated, teams build with confidence. When policies travel with the code, security reviews become approvals instead of interrogations. When compliance is infrastructure rather than inspection, pilots graduate to production in weeks, not quarters.

The goal is to build the scaffolding that lets you move fast because you're safe.

What This Cookbook Delivers

This guide shows you how to make governance part of core infrastructure from day one, instead of a launch-time afterthought.

You'll learn to:

  • Define policies as code that version, travel, and deploy alongside your applications
  • Apply guardrails automatically to every AI call - no manual review bottlenecks
  • Evaluate your defenses with precision and recall metrics, so you know they actually work
  • Package governance for distribution so any team can pip install instant compliance
  • Build agentic systems with proper handoffs, observability, and oversight from day one

Each section pairs a concrete governance objective with a practical implementation pattern, including example configurations, code snippets, and integration points you can adapt to your own environment.

By the end, you'll have a working blueprint for governed AI that scales across your organization and turns governance from a friction point into a competitive advantage.


What We'll Build

We'll create a Private Equity firm AI assistant with:

  1. Multiple specialist agents that handle different domains
  2. A triage agent that routes queries via handoffs
  3. Built-in guardrails that validate queries before processing
  4. Tracing for full observability of agent behavior
  5. Centralized policy enforcement via an installable package
  6. Eval-driven system design for reliability & scalability

The architecture looks like this:

The pipeline treats red team (adversarial) inputs the same way as user queries: they flow through pre-flight, input guardrails, orchestration, and output guardrails. GuardrailEval and the feedback loop use results from both normal and adversarial runs to tune policy and harden defenses.

Prerequisites

Before we begin, you'll need:

  • Python 3.9+
  • An OpenAI API key
  • A GitHub account (for the policy repo)

Let's set up our environment.


Building the System

In this section we'll build a PE firm AI assistant from scratch: define tools, create specialist agents, and wire up handoffs between them.

Understanding Agents and Tools

An agent is an AI system that can:

  • Receive instructions that define its role and behavior
  • Use tools to take actions (search databases, create records, call APIs)
  • Hand off to other agents when a task is outside its expertise
  • Maintain context across a conversation

Think of agents like employees with specific job descriptions. A receptionist (triage agent) knows who to route calls to, while specialists (domain agents) have deep expertise in specific areas.

Why Use Tools?

Tools extend what agents can do beyond just generating text:

Without Tools With Tools
"I can tell you about deal evaluation best practices" "Let me search your deal database... Found 3 matches"
"You should check your portfolio metrics" "Acme Corp: Revenue $50M (+15% YoY), EBITDA $8M"
"Consider creating a deal memo" "Deal memo created for TechCorp in your system"

Important: OpenAI doesn't execute tools for you - it tells your application which tools to call and with what parameters. Your code executes the actual logic.

Step 1: Define Tools

Tools are Python functions decorated with @function_tool. The docstring becomes the tool's description that the agent sees.


Multi-Agent System with Handoffs

Real-world tasks rarely fit into a single agent's expertise. Consider a PE firm:

  • Deal questions need investment criteria knowledge
  • Portfolio questions need operational metrics expertise
  • LP questions need compliance awareness and fund knowledge

You could build one massive agent with all this knowledge, but instructions become unwieldy, the agent struggles to stay "in character", and you can't easily update one domain without affecting others.

Handoffs solve this by letting agents delegate to specialists:

User: "What's our IRR on Fund II?"
    │
    ▼
Triage Agent: "This is an LP/investor question"
    │
    ▼ (handoff)
IR Agent: "Fund II IRR is 22.5% net as of Q3..."

The user sees one seamless conversation, but behind the scenes, the right expert is answering.

Step 2: Create Specialist Agents

Each specialist has:

  • name: Identifier for the agent
  • handoff_description: Tells the triage agent WHEN to route here (critical!)
  • instructions: Defines HOW the agent should behave

Step 3: Create the Triage Agent

The triage agent is the "front door". It:

  1. Receives all incoming queries
  2. Decides which specialist should handle it (using handoff_description)
  3. Hands off the conversation seamlessly

The handoffs parameter tells the agent which specialists it can delegate to.


Basic Observability & Guardrails

With the agent system built, we now add observability (tracing) and basic guardrails to make it production-ready.

Tracing - Observability for Agents

With multi-agent systems, a single user query can trigger multiple LLM calls, tool executions, handoffs between agents, and guardrail checks. Tracing captures all of this in a structured way, giving you:

Benefit Description
Debugging See exactly what happened when something goes wrong
Performance Identify slow steps in your agent workflows
Auditing Review what agents did and why
Optimization Find opportunities to improve prompts or reduce calls

Using the trace() Context Manager

The trace() function wraps operations under a named trace, linking all spans together. After running, you can view the complete trace - including every LLM call, tool execution, and handoff - in the OpenAI Traces Dashboard.

Trace Naming Best Practices

Good trace names help you find and analyze specific workflows:

### ❌ Bad: Generic names
with trace("query"):
    ...

### ✅ Good: Descriptive, searchable names
with trace("Deal Screening - Healthcare"):
    ...

with trace(f"LP Inquiry - {lp_name}"):
    ...

with trace(f"Portfolio Review - {company} - Q{quarter}"):
    ...

Tracing for Compliant Industries (Zero Data Retention)

Some organizations have Zero Data Retention (ZDR) agreements with OpenAI, meaning:

  • Data is not stored or retained after processing
  • The built-in tracing dashboard cannot be used (it stores traces in OpenAI's systems)

This is common in financial services, healthcare (HIPAA), government, and organizations with strict data residency rules.

Org Type Built-in Dashboard What to Do
Non-ZDR ✅ Allowed Use default tracing; view traces in dashboard
ZDR (strict) ❌ Not allowed Disable tracing entirely
ZDR (needs observability) ❌ Not allowed Use trace processors to stream to your internal systems

Option 1: Disable Tracing Entirely

For strict ZDR compliance, disable tracing globally or per-run.

Option 2: Custom Trace Processors (Internal Observability)

If you need observability but can't use OpenAI's dashboard, you can export traces to your own systems.

This keeps traces:

  • Within your infrastructure
  • Under your data retention policies
  • Integrated with your existing monitoring stack

Best Practices for ZDR Tracing

  1. Use trace processors to maintain visibility while keeping data internal
  2. Redact PII in your processor before storing spans
  3. Set retention policies that match your compliance requirements
  4. Audit access to trace data in your internal systems
  5. Document your approach for compliance reviews

Adding Built-in Guardrails

The Agents SDK has built-in guardrails that run at the agent level. These are useful for agent-specific validation.

Let's add a guardrail that ensures queries are relevant to PE operations.


Centralizing Governance

Built-in guardrails are great, but they require configuration on each agent. For organization-wide governance, we want to:

  1. Define policy once in a central location
  2. Apply automatically to all OpenAI calls
  3. Version control the policy like code
  4. Install via pip in any project

This is where the OpenAI Guardrails library comes in.

Centralized Policy with OpenAI Guardrails

Aspect Built-in (Agents SDK) Centralized (Guardrails Library)
Scope Per-agent All OpenAI calls
Configuration In code, per agent JSON config, org-wide
Best for Domain-specific rules Universal policies
Example "Is this a PE question?" "Block prompt injection everywhere"

Available Guardrails

Creating a Policy Config

The config has two stages:

  • input: Runs BEFORE the LLM call (block bad inputs)
  • output: Runs AFTER the LLM response (redact sensitive outputs)

💡 Tip: Use the OpenAI Guardrails Wizard

Instead of writing the config JSON by hand, you can use the OpenAI Guardrails Wizard to:

  1. Select guardrails from an interactive UI (PII detection, moderation, prompt injection, etc.)
  2. Configure thresholds and categories visually
  3. Export the config JSON and integration code directly

This is the fastest way to generate a production-ready policy config. The wizard produces the same JSON format used below - you can paste it directly into your policy package.

Using GuardrailsOpenAI

The GuardrailsOpenAI client wraps the standard OpenAI client and automatically applies guardrails.


Creating a Reusable Policy Package

Package your policy for organization-wide use. Any team can:

pip install git+https://github.com/yourorg/policies.git

And immediately have governance:

from your_policies import GUARDRAILS_CONFIG
client = GuardrailsOpenAI(config=GUARDRAILS_CONFIG)
### All calls are now governed!

Key benefits: consistency across projects, easy updates via pip upgrade, full audit trail via Git history, and a single compliance reference point.

Step-by-Step: Creating the Policy Repo

1. Create a new GitHub repository
mkdir pe-policies
cd pe-policies
git init
2. Create the package structure
pe-policies/
├── pe_policies/
│   ├── __init__.py      # Exports GUARDRAILS_CONFIG
│   └── config.json      # The actual guardrails config
├── pyproject.toml       # Package metadata
├── README.md            # Documentation
└── POLICY.md            # Human-readable policy document
3. Create pe_policies/__init__.py
import json
from pathlib import Path

_config_path = Path(__file__).parent / "config.json"

with open(_config_path) as f:
    GUARDRAILS_CONFIG = json.load(f)

__all__ = ["GUARDRAILS_CONFIG"]
4. Create pe_policies/config.json

Use the same policy structure defined in PE_FIRM_POLICY above. Here's a condensed view:

{
  "version": 1,
  "pre_flight": {
    "version": 1,
    "guardrails": [
      { "name": "Contains PII", "config": { "entities": ["CREDIT_CARD", "EMAIL_ADDRESS", "US_SSN", "..." ], "block": true }},
      { "name": "Moderation", "config": { "categories": ["sexual", "hate", "violence", "..."] }}
    ]
  },
  "input": {
    "version": 1,
    "guardrails": [
      { "name": "Jailbreak", "config": { "confidence_threshold": 0.7, "model": "gpt-4.1-mini" }},
      { "name": "Off Topic Prompts", "config": { "confidence_threshold": 0.7, "model": "gpt-4.1-mini", "system_prompt_details": "..." }}
    ]
  },
  "output": {
    "version": 1,
    "guardrails": [
      { "name": "Contains PII", "config": { "entities": ["CREDIT_CARD", "EMAIL_ADDRESS", "..."], "block": true }}
    ]
  }
}

See PE_FIRM_POLICY in the Centralized Policy section for the full configuration with all entities and categories.

Note: The "block": true setting is required for the PII guardrail in the output stage. Without it, PII will be detected and masked but won't trigger a block.

5. Create pyproject.toml
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "pe-policies"
version = "0.1.0"
description = "PE Firm AI Agent Policy Configuration"
requires-python = ">=3.9"
dependencies = []

[tool.setuptools.packages.find]
include = ["pe_policies*"]

[tool.setuptools.package-data]
pe_policies = ["*.json"]
6. Push to GitHub
git add .
git commit -m "Initial policy package"
git remote add origin https://github.com/yourorg/pe-policies.git
git push -u origin main
7. Install and use from any project
pip install git+https://github.com/yourorg/pe-policies.git
from pe_policies import GUARDRAILS_CONFIG
from guardrails import GuardrailsOpenAI

client = GuardrailsOpenAI(config=GUARDRAILS_CONFIG)
### All calls now have governance automatically applied!

Putting It All Together

Here's the complete pattern for a governed agent system:


Improving & Optimizing

With the governed system running, we now evaluate, tune, and stress-test it.

Evaluating Your Guardrails

Building guardrails is only half the battle - you need to know they actually work. The OpenAI Guardrails library includes a built-in evaluation framework that measures precision, recall, and F1 scores against labeled test data.

Metric What It Measures Why It Matters
Precision Of all blocked queries, how many should have been blocked? High precision = few false positives (legitimate queries blocked)
Recall Of all bad queries, how many did we catch? High recall = few false negatives (threats getting through)
F1 Score Harmonic mean of precision and recall Balanced measure of overall performance

The trade-off: high precision with low recall means threats slip through; high recall with low precision blocks legitimate queries. Adjust confidence_threshold to find the right balance.

Step 1: Load the Test Dataset

The evaluation framework expects a JSONL file where each line contains:

  • id: Unique identifier for the test case
  • data: The input text (plain string or multi-turn JSON)
  • expected_triggers: Dict mapping each guardrail name to true/false

The full dataset (21 samples covering PII, moderation, jailbreak, off-topic, and mixed cases) is in eval_data/guardrail_test_data.jsonl. Below we load it and inspect the coverage.

Step 2: Create the Eval Config

We use PE_FIRM_POLICY directly as the eval config - evaluate what you deploy. This covers all three stages: pre-flight (PII, Moderation), input (Jailbreak, Off Topic), and output (PII).

Step 3: Run the Evaluation

You can run evals via CLI or programmatically. Here's both approaches:

Eval Best Practices

  1. Build diverse test sets: Include edge cases, adversarial examples, and legitimate queries
  2. Balance your dataset: Ensure roughly equal positive and negative examples per guardrail
  3. Run evals on policy changes: Before deploying updated confidence_threshold values
  4. Benchmark across models: Use --mode benchmark to compare gpt-5.2-mini vs gpt-5.2 for LLM-based guardrails
  5. Automate in CI/CD: Run evals on every policy repo change to catch regressions
### Benchmark mode compares models and generates ROC curves
guardrails-evals \
  --config-path config.json \
  --dataset-path test_data.jsonl \
  --mode benchmark \
  --models gpt-5.2-mini gpt-5.2

Automated Feedback Loop for Threshold Tuning

Manually tuning confidence_threshold values based on eval results is tedious. The Guardrail Feedback Loop automates this: it runs evals, analyzes precision/recall gaps, adjusts thresholds, re-validates, and saves the tuned config when metrics improve.

The loop includes oscillation prevention - if threshold adjustments keep flip-flopping, it reduces step size and eventually stops tuning that guardrail.

Step 1: Create a Tunable Configuration

We derive the tunable config directly from PE_FIRM_POLICY - the same config our GuardrailAgent uses - so we're tuning the actual production guardrails. The only change is overriding confidence_threshold values to an intentionally high starting point.

LLM-based guardrails like Jailbreak and Off Topic Prompts use confidence thresholds to decide when to trigger. The threshold controls the trade-off:

  • Higher threshold (e.g., 0.95): More conservative, fewer false positives, but may miss some threats
  • Lower threshold (e.g., 0.5): More sensitive, catches more threats, but may block legitimate queries

For this demo, we'll start with an intentionally high threshold (0.95) so you can see the tuner detect low recall and automatically decrease it.

Step 2: Create a Test Dataset

The feedback loop needs labeled test data to measure guardrail performance. Each test case specifies:

  • data: The text to evaluate - for conversation-aware guardrails (Jailbreak, Prompt Injection), we use multi-turn format with the agent's system prompt included so the guardrail evaluates in the same context as production
  • expected_triggers: Which guardrails should fire (true) or not (false)

Include both positive examples (should trigger) and negative examples (should not trigger) for precision/recall measurement. We also include adversarial PE-domain-camouflaged attacks and borderline-but-legitimate queries to stress-test false positive/negative rates.

Step 3: Run the Feedback Loop

Now we run the automated tuning process. The GuardrailFeedbackLoop will:

  1. Run an initial evaluation to get baseline metrics
  2. Compare precision/recall against our targets (90% each)
  3. Adjust thresholds based on which metric is underperforming
  4. Re-run evals to measure the impact
  5. Repeat until targets are met or max iterations reached

What to expect: With our intentionally high threshold (0.95), the initial eval will show low recall (the guardrail misses some jailbreak attempts). The tuner will detect this and decrease the threshold until recall meets the 90% target.

Watch the logs to see the loop's decision-making in action.

Step 4: Review the Results

After tuning completes, we can inspect what changes were made:

  • Threshold changes: How the confidence_threshold was adjusted
  • Metric improvements: Changes in precision, recall, and F1 score
  • Convergence status: Whether targets were achieved or tuning stopped early

The tuned configuration is automatically saved for use in production.

CLI Usage

You can also run the feedback loop from the command line:

### Basic usage
python tune_guardrails.py \
    --config eval_data/tunable_config.json \
    --dataset eval_data/input_guardrail_test_data.jsonl \
    --output tuning_results

### With custom targets
python tune_guardrails.py \
    --config eval_data/tunable_config.json \
    --dataset eval_data/input_guardrail_test_data.jsonl \
    --precision-target 0.95 \
    --recall-target 0.85 \
    --priority precision \
    --max-iterations 15 \
    --verbose

Output files include tuning_results/eval_config_tuned.json (optimized config), tuning_results/tuning_report_*.md (detailed report), and backups of original configs.


Red Teaming Your Guardrails with Promptfoo

Evals measured guardrail detection accuracy - "Did the guardrail fire correctly on known test cases?" But there's a harder question: "Can an attacker bypass your guardrails?"

Promptfoo is an open-source red teaming tool that auto-generates hundreds of adversarial inputs across 50+ vulnerability types - jailbreaks, prompt injections, PII extraction, off-topic hijacking, and more. Instead of writing test cases by hand, Promptfoo creates sophisticated, adaptive attacks and tests them against your actual application.

OpenAI Guardrails Eval Promptfoo Red Team
Tests guardrail detection accuracy (precision/recall) Tests whether adversarial inputs bypass guardrails
You write test cases manually Auto-generates hundreds of adversarial cases
Static dataset Adaptive attacks that evolve based on responses
"Did the guardrail fire?" "Can an attacker get through?"

Together they form a complete testing strategy: guardrails eval ensures detection quality, Promptfoo ensures resilience against real-world attacks.

How It Works Under the Hood

Promptfoo uses your existing OPENAI_API_KEY to power a three-phase process:

Your OPENAI_API_KEY
       │
       ▼
┌──────────────┐    adversarial     ┌──────────────────┐
│   Promptfoo   │─── prompts ──────▶│  Your target.py   │
│   (attacker)  │                   │  (GuardrailAgent) │
│   LLM generates◀── responses ────│                   │
│   & grades    │                   └──────────────────┘
└──────────────┘
       │
       ▼
  Red Team Report
  1. Generate: An LLM (defaults to gpt-5) generates adversarial prompts tailored to your application's purpose and selected plugins
  2. Attack: Each generated prompt is sent to your Python target script, which runs it through the governed agent (Runner.run)
  3. Grade: Another LLM call evaluates whether the response indicates a successful bypass or a proper block

Prerequisites and Cost

  • Promptfoo: Free, open source (MIT license)
  • Email verification: One-time free email check on first run (spam prevention, not a subscription)
  • LLM cost: Your standard OpenAI API usage for attack generation + grading. With numTests: 10 across ~9 plugins, expect ~100-200 API calls (a few dollars)
  • No subscription required -- your existing OPENAI_API_KEY is all you need

Step 1: Install Promptfoo

pip install promptfoo

Note: The pip package is a lightweight wrapper that requires Node.js 20+ installed on your system. Install Node via brew install node (macOS), sudo apt install nodejs npm (Ubuntu), or from nodejs.org.

Step 2: The Target Script

Promptfoo needs a way to talk to your governed agent. The file promptfoo/promptfoo_target.py bridges Promptfoo to your GuardrailAgent:

  • Receives each adversarial prompt from Promptfoo
  • Runs it through Runner.run(pe_concierge_governed, prompt) - the full agent with handoffs, tools, and centralized guardrails
  • Returns the response, or [BLOCKED] if any guardrail fires

The script recreates the same agent stack from the notebook: specialist agents, tools, custom pe_guardrail, PE_FIRM_POLICY, and the GuardrailAgent triage agent.

Step 3: The Red Team Config

The file promptfoo/promptfooconfig.yaml defines what to attack and how:

targets:
 - id: "python:promptfoo/promptfoo_target.py"
    label: "pe-concierge-governed"

purpose: >  # Application context improves attack quality
  A Private Equity firm front-desk AI assistant that handles deal screening,
  portfolio management, and investor relations...

redteam:
  numTests: 10  # Adversarial inputs per plugin
  plugins:                     # Generate adversarial inputs
   - hijacking                # Off-topic hijacking
   - pii:direct               # PII extraction attempts
   - prompt-extraction        # System prompt extraction
   - system-prompt-override   # Override system instructions
   - off-topic                # Off-topic manipulation
   - policy                   # Custom policy violations
  strategies:                  # Wrap inputs in evasion techniques
   - jailbreak                # Jailbreak wrapper patterns
   - prompt-injection         # Injection wrapper patterns
   - base64                   # Base64 encoding evasion
   - leetspeak                # l33tspeak encoding
   - rot13                    # ROT13 encoding evasion
   - crescendo                # Gradually escalating attacks

Plugins generate adversarial inputs targeting specific vulnerabilities. Strategies wrap those inputs in evasion techniques (jailbreak patterns, encoding, translation) to test whether guardrails can be bypassed beyond simple text matching. See the full plugin list for 131 available plugins.

Step 4: Run the Red Team

### Navigate to the promptfoo directory
cd promptfoo

### Generate adversarial inputs and run them against your agent
promptfoo redteam run

### View the interactive report
promptfoo redteam report

The report shows:

  • Pass/fail rate per vulnerability category
  • Severity levels for each finding
  • Concrete examples of inputs that bypassed guardrails
  • Suggested mitigations for each vulnerability

Sample Report

Here's what a successful red team report looks like -- 0 vulnerabilities across all categories, 33/33 tests defended:

The report breaks results into Risk Categories (Security & Access Control, Brand) and individual tests (Resource Hijacking, System Prompt Override, PII via Direct Exposure, Off-Topic Manipulation). Our GuardrailAgent with PE_FIRM_POLICY blocked 100% of the adversarial inputs.

Going Deeper

This demo used 5 plugins with numTests: 3 for a quick 33-probe scan. For production-grade assessments, increase the depth to 50+ probes per plugin and enable preset collections like owasp:llm (OWASP LLM Top 10), nist:ai:measure (NIST AI RMF), or mitre:atlas -- Promptfoo supports 131 plugins across security, compliance, trust & safety, and brand categories.

Interpreting Results

Any failures reveal gaps in your PE_FIRM_POLICY that need attention -- whether that's lowering thresholds, adding guardrails, or refining system prompts.

CI/CD Integration

Add red teaming to your deployment pipeline so guardrail changes are validated automatically:

### .github/workflows/redteam.yml
name: Red Team Guardrails
on:
  push:
    paths: ['guardrails/**']
jobs:
  redteam:
    runs-on: ubuntu-latest
    steps:
     - uses: actions/checkout@v4
     - uses: actions/setup-node@v4
        with: { node-version: 20 }
     - run: pip install promptfoo
     - run: promptfoo redteam run
     - run: promptfoo redteam report --output redteam-report.html
     - uses: actions/upload-artifact@v4
        with:
          name: redteam-report
          path: redteam-report.html

Key Takeaways

1. Governance enables adoption

By establishing clear guardrails upfront, you remove the fear and uncertainty that slows AI adoption. Teams can build confidently knowing policies are enforced automatically. Governance becomes an execution system that keeps adoption moving securely and at scale.

2. Use handoffs for specialization

Avoid one massive agent. Create specialists and let them collaborate. The handoff_description is key to good routing.

3. Layer your defenses

  • OpenAI Guardrails (client-level): Universal policies for all calls
  • Agents SDK guardrails (agent-level): Domain-specific validation

4. Trace everything (or nothing, for ZDR)

  • Use trace() to group operations for debugging
  • For ZDR compliance: disable tracing or use custom processors

5. Centralize policy, distribute capability

The policy-as-a-package pattern lets you:

  • Maintain governance in one place
  • Update policies without changing application code
  • Audit compliance across all projects

Next Steps

Initial Setup

  1. Create your policy repo using the template above
  2. Customize guardrails for your industry and compliance requirements
  3. Add custom trace processors if you need ZDR-compliant observability
  4. Document your policy alongside the code
  5. Set up CI/CD to test policy changes before deployment

Scaling AI Across Your Organization

When moving from prototype to production, consider how different user groups will interact with AI:

Role What They Build Governance Approach
Developers Custom agents, MCP connectors, integrations Safe defaults, reusable templates, evaluation pipelines
Power Users Configured assistants, automated workflows Pre-approved patterns, governed portals
End Users Content generation, data analysis Curated tools with embedded guardrails

This approach ensures everyone, from engineers to analysts, can leverage AI safely within appropriate boundaries.

Enabling Citizen Developers

Empower non-technical teams to build safely:

  • Provide templates for prompt packs, tool configurations, and evaluation checks
  • Create review lanes and publishing workflows that make it easy to build and deploy
  • Offer guardrailed sandboxes for experimentation without risking sensitive data
  • Establish clear promotion paths from prototype to production with governance checkpoints

Registries for Visibility

Treat AI assets as first-class governed resources by maintaining registries:

  • Agent Registry: Register all agents with owner, purpose, risk tier, and evaluation status
  • Tool Registry: Document MCP tools with authentication scopes, data access, and approval authority
  • Prompt Registry: Version and govern prompts like code, with lineage, rollback policies, and change controls

Registry metadata enables discoverability, auditing, and lifecycle management across your AI ecosystem.

Risk-Proportionate Controls

Not all AI use cases carry the same risk. Differentiate your controls:

  • Low-risk (internal productivity, non-sensitive data): Fast-track approval, minimal logging
  • Moderate-risk (customer-facing, operational data): Standard guardrails, audit trails
  • High-risk (PII, financial, regulated): Enhanced logging, human-in-the-loop, isolated environments

Apply proportionate controls, approvals, review, and detailed logging, only where necessary, keeping lightweight adoption fast and frictionless.

Preventing Shadow AI

Centralized governance helps prevent unauthorized AI tools from proliferating:

  • Make governed options easier than ungoverned alternatives
  • Provide clear adoption paths for different skill levels and use cases
  • Incorporate discovery mechanisms to detect and catalog unsanctioned AI activity
  • Offer support and training so teams don't go around the system

Early visibility allows governance teams to close gaps before they become systemic risks.

Standards Alignment

Align your governance practices with recognized frameworks:

  • NIST AI RMF - Risk management framework for AI systems
  • ISO/IEC 42001 - AI management system standard
  • Industry-specific requirements (HIPAA, SOX, GDPR, etc.)

Building on established standards creates external credibility alongside internal control.


Resources


FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.