Prompt Chain

Build Governed AI Agents with Scaffolding

Build governed multi-agent AI systems with policy-as-code guardrails, pip-installable compliance packages, and eval-driven hardening.

Works with openai

76
Spark score
out of 100
Updated last month
Version 1.0.0
Models
gpt 4o

Add to Favorites

Why it matters

Implement a robust framework for building and deploying AI agents safely and scalably within an organization, ensuring governance is integrated from the start.

Outcomes

What it gets done

01

Define and enforce policies as code for AI applications.

02

Automate guardrail application to AI calls for security and compliance.

03

Develop multi-agent systems with clear handoffs and observability.

04

Package governance for easy distribution and integration across teams.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/oai-agenticgovernancecookbook | bash

Steps

Steps in the chain

01
Step 1: Define Tools
02
Step 2: Create Specialist Agents
03
Step 3: Create the Triage Agent

Overview

Building Governed AI Agents: A Practical Guide to Agentic Scaffolding

Builds a governed multi-agent PE-firm assistant with policy-as-code guardrails, pip-installable compliance packaging, and precision/recall-evaluated defenses. Use when moving AI agent pilots into production and needing auditable, automated governance shared consistently across teams.

What it does

This cookbook shows how to make governance part of an AI agent system's core infrastructure from day one, rather than a launch-time afterthought, using a Private Equity firm AI assistant as the worked example. The architecture combines: multiple specialist agents for different domains (deal evaluation, portfolio metrics, LP/compliance questions), a triage agent that routes queries to the right specialist via handoffs (delegation, since one massive agent covering all domains becomes unwieldy and hard to maintain), Python function tools (decorated with @function_tool, docstring as the tool description) that let agents take real actions like searching a deal database or creating records rather than just generating text, built-in guardrails validating queries at pre-flight, input, and output stages, full tracing/observability of agent behavior, and centralized policy enforcement distributed as an installable pip package.

Governance is defined as versioned JSON policy configuration - e.g. a pre-flight PII/moderation check (blocking on entities like CREDIT_CARD, EMAIL_ADDRESS, US_SSN), input guardrails for jailbreak and off-topic-prompt detection (using a confidence threshold and a lightweight model like gpt-4.1-mini as the classifier), and output guardrails re-checking for PII (noting the "block": true setting is required, since without it PII is only masked, not blocked). Critically, the same guardrail pipeline processes both real user queries and red-team/adversarial inputs identically - pre-flight, input guardrails, orchestration, output guardrails - and a GuardrailEval component uses results from both normal and adversarial runs to tune policy and harden defenses, measured with precision/recall metrics so teams know their guardrails actually work rather than assuming so.

The policy itself is packaged as a standalone Python package (pe_policies, with __init__.py loading a config.json, a pyproject.toml for packaging, pushed to GitHub) so any project can pip install git+https://github.com/yourorg/pe-policies.git and get from pe_policies import GUARDRAILS_CONFIG - instant, versioned, shareable compliance that travels with the code rather than being re-implemented per project.

When to use - and when NOT to

Use this pattern when an organization needs to move AI pilots into production with real customer/data exposure and needs concrete, auditable answers to "what happens when it fails," "who's accountable," and "how do we prove compliance" - not just a working demo. The core insight is that automated, code-based guardrails accelerate rather than slow delivery, since security reviews become approvals of an already-governed system rather than ad hoc audits. Use the pip-installable policy package pattern specifically when multiple teams/agents need the same compliance rules applied consistently without duplicating configuration.

Inputs and outputs

Input: user (or adversarial/red-team) queries flowing through pre-flight checks, input guardrails, multi-agent orchestration, and output guardrails. Output: either a governed agent response (with PII masked/blocked and off-topic/jailbreak attempts refused per policy) or a blocked/refused response, plus eval metrics (precision/recall) on how well the guardrails actually catch violations.

Integrations

Requires Python 3.9+, an OpenAI API key, a GitHub account for the policy repo, the OpenAI Agents SDK (@function_tool, handoffs), a guardrails library (GuardrailsOpenAI), and gpt-4.1-mini as the guardrail classifier model.

from pe_policies import GUARDRAILS_CONFIG
from guardrails import GuardrailsOpenAI
client = GuardrailsOpenAI(config=GUARDRAILS_CONFIG)

Who it's for

Organizations moving AI agent pilots into production who need policy-as-code guardrails, multi-agent handoffs, and measurable, shareable compliance rather than manual per-launch security review.

Source README

Building Governed AI Agents: A Practical Guide to Agentic Scaffolding

A cookbook for enabling safe, scalable AI agent adoption in your organization


The Shift in Mindset

Every enterprise faces the same tension: the pressure to adopt AI is immense, but so is the fear of getting it wrong. Teams want to build, legal wants to review, security wants to audit, and promising pilots stall because no one can answer: "Is this safe to deploy?"

Organizations have moved past "Should we experiment with AI?" and now ask "How do we get this into production safely?" The prototypes worked, the demos impressed the board, and now there's real pressure to deliver AI that touches real customers and handles real data. But production demands answers that pilots never required: What happens when it fails? Who's accountable? How do we prove it's compliant?

The organizations winning at AI have discovered something counterintuitive: governance drives delivery. When guardrails are clear and automated, teams build with confidence. When policies travel with the code, security reviews become approvals instead of interrogations. When compliance is infrastructure rather than inspection, pilots graduate to production in weeks, not quarters.

The goal is to build the scaffolding that lets you move fast because you're safe.

What This Cookbook Delivers

This guide shows you how to make governance part of core infrastructure from day one, instead of a launch-time afterthought.

You'll learn to:

  • Define policies as code that version, travel, and deploy alongside your applications
  • Apply guardrails automatically to every AI call - no manual review bottlenecks
  • Evaluate your defenses with precision and recall metrics, so you know they actually work
  • Package governance for distribution so any team can pip install instant compliance
  • Build agentic systems with proper handoffs, observability, and oversight from day one

Each section pairs a concrete governance objective with a practical implementation pattern, including example configurations, code snippets, and integration points you can adapt to your own environment.

By the end, you'll have a working blueprint for governed AI that scales across your organization and turns governance from a friction point into a competitive advantage.


What We'll Build

We'll create a Private Equity firm AI assistant with:

  1. Multiple specialist agents that handle different domains
  2. A triage agent that routes queries via handoffs
  3. Built-in guardrails that validate queries before processing
  4. Tracing for full observability of agent behavior
  5. Centralized policy enforcement via an installable package
  6. Eval-driven system design for reliability & scalability

The architecture looks like this:

The pipeline treats red team (adversarial) inputs the same way as user queries: they flow through pre-flight, input guardrails, orchestration, and output guardrails. GuardrailEval and the feedback loop use results from both normal and adversarial runs to tune policy and harden defenses.

Prerequisites

Before we begin, you'll need:

  • Python 3.9+
  • An OpenAI API key
  • A GitHub account (for the policy repo)

Let's set up our environment.


Building the System

In this section we'll build a PE firm AI assistant from scratch: define tools, create specialist agents, and wire up handoffs between them.

Understanding Agents and Tools

An agent is an AI system that can:

  • Receive instructions that define its role and behavior
  • Use tools to take actions (search databases, create records, call APIs)
  • Hand off to other agents when a task is outside its expertise
  • Maintain context across a conversation

Think of agents like employees with specific job descriptions. A receptionist (triage agent) knows who to route calls to, while specialists (domain agents) have deep expertise in specific areas.

Why Use Tools?

Tools extend what agents can do beyond just generating text:

Without Tools With Tools
"I can tell you about deal evaluation best practices" "Let me search your deal database... Found 3 matches"
"You should check your portfolio metrics" "Acme Corp: Revenue $50M (+15% YoY), EBITDA $8M"
"Consider creating a deal memo" "Deal memo created for TechCorp in your system"

Important: OpenAI doesn't execute tools for you - it tells your application which tools to call and with what parameters. Your code executes the actual logic.

Step 1: Define Tools

Tools are Python functions decorated with @function_tool. The docstring becomes the tool's description that the agent sees.


Multi-Agent System with Handoffs

Real-world tasks rarely fit into a single agent's expertise. Consider a PE firm:

  • Deal questions need investment criteria knowledge
  • Portfolio questions need operational metrics expertise
  • LP questions need compliance awareness and fund knowledge

You could build one massive agent with all this knowledge, but instructions become unwieldy, the agent struggles to stay "in character", and you can't easily update one domain without affecting others.

Handoffs solve this by letting agents delegate to specialists:

User: "What's our IRR on Fund II?"
    │
    ▼
Triage Agent: "This is an LP/investor question"
    │
    ▼ (handoff)
IR Agent: "Fund II IRR is 22.5% net as of Q3..."

The user sees one seamless conversation, but behind the scenes, the right expert is answering.

Step 2: Create Specialist Agents

Each specialist has:

  • name: Identifier for the agent
  • handoff_description: Tells the triage agent WHEN to route here (critical!)
  • instructions: Defines HOW the agent should behave

Step 3: Create the Triage Agent

The triage agent is the "front door". It:

  1. Receives all incoming queries
  2. Decides which specialist should handle it (using handoff_description)
  3. Hands off the conversation seamlessly

The handoffs parameter tells the agent which specialists it can delegate to.


Basic Observability & Guardrails

With the agent system built, we now add observability (tracing) and basic guardrails to make it production-ready.

Tracing - Observability for Agents

With multi-agent systems, a single user query can trigger multiple LLM calls, tool executions, handoffs between agents, and guardrail checks. Tracing captures all of this in a structured way, giving you:

Benefit Description
Debugging See exactly what happened when something goes wrong
Performance Identify slow steps in your agent workflows
Auditing Review what agents did and why
Optimization Find opportunities to improve prompts or reduce calls

Using the trace() Context Manager

The trace() function wraps operations under a named trace, linking all spans together. After running, you can view the complete trace - including every LLM call, tool execution, and handoff - in the OpenAI Traces Dashboard.

Trace Naming Best Practices

Good trace names help you find and analyze specific workflows:

### ❌ Bad: Generic names
with trace("query"):
    ...

### ✅ Good: Descriptive, searchable names
with trace("Deal Screening - Healthcare"):
    ...

with trace(f"LP Inquiry - {lp_name}"):
    ...

with trace(f"Portfolio Review - {company} - Q{quarter}"):
    ...

Tracing for Compliant Industries (Zero Data Retention)

Some organizations have Zero Data Retention (ZDR) agreements with OpenAI, meaning:

  • Data is not stored or retained after processing
  • The built-in tracing dashboard cannot be used (it stores traces in OpenAI's systems)

This is common in financial services, healthcare (HIPAA), government, and organizations with strict data residency rules.

Org Type Built-in Dashboard What to Do
Non-ZDR ✅ Allowed Use default tracing; view traces in dashboard
ZDR (strict) ❌ Not allowed Disable tracing entirely
ZDR (needs observability) ❌ Not allowed Use trace processors to stream to your internal systems

Option 1: Disable Tracing Entirely

For strict ZDR compliance, disable tracing globally or per-run.

Option 2: Custom Trace Processors (Internal Observability)

If you need observability but can't use OpenAI's dashboard, you can export traces to your own systems.

This keeps traces:

  • Within your infrastructure
  • Under your data retention policies
  • Integrated with your existing monitoring stack

Best Practices for ZDR Tracing

  1. Use trace processors to maintain visibility while keeping data internal
  2. Redact PII in your processor before storing spans
  3. Set retention policies that match your compliance requirements
  4. Audit access to trace data in your internal systems
  5. Document your approach for compliance reviews

Adding Built-in Guardrails

The Agents SDK has built-in guardrails that run at the agent level. These are useful for agent-specific validation.

Let's add a guardrail that ensures queries are relevant to PE operations.


Centralizing Governance

Built-in guardrails are great, but they require configuration on each agent. For organization-wide governance, we want to:

  1. Define policy once in a central location
  2. Apply automatically to all OpenAI calls
  3. Version control the policy like code
  4. Install via pip in any project

This is where the OpenAI Guardrails library comes in.

Centralized Policy with OpenAI Guardrails

Aspect Built-in (Agents SDK) Centralized (Guardrails Library)
Scope Per-agent All OpenAI calls
Configuration In code, per agent JSON config, org-wide
Best for Domain-specific rules Universal policies
Example "Is this a PE question?" "Block prompt injection everywhere"

Available Guardrails

Creating a Policy Config

The config has two stages:

  • input: Runs BEFORE the LLM call (block bad inputs)
  • output: Runs AFTER the LLM response (redact sensitive outputs)

💡 Tip: Use the OpenAI Guardrails Wizard

Instead of writing the config JSON by hand, you can use the OpenAI Guardrails Wizard to:

  1. Select guardrails from an interactive UI (PII detection, moderation, prompt injection, etc.)
  2. Configure thresholds and categories visually
  3. Export the config JSON and integration code directly

This is the fastest way to generate a production-ready policy config. The wizard produces the same JSON format used below - you can paste it directly into your policy package.

Using GuardrailsOpenAI

The GuardrailsOpenAI client wraps the standard OpenAI client and automatically applies guardrails.


Creating a Reusable Policy Package

Package your policy for organization-wide use. Any team can:

pip install git+https://github.com/yourorg/policies.git

And immediately have governance:

from your_policies import GUARDRAILS_CONFIG
client = GuardrailsOpenAI(config=GUARDRAILS_CONFIG)
### All calls are now governed!

Key benefits: consistency across projects, easy updates via pip upgrade, full audit trail via Git history, and a single compliance reference point.

Step-by-Step: Creating the Policy Repo

1. Create a new GitHub repository
mkdir pe-policies
cd pe-policies
git init
2. Create the package structure
pe-policies/
├── pe_policies/
│   ├── __init__.py      # Exports GUARDRAILS_CONFIG
│   └── config.json      # The actual guardrails config
├── pyproject.toml       # Package metadata
├── README.md            # Documentation
└── POLICY.md            # Human-readable policy document
3. Create pe_policies/__init__.py
import json
from pathlib import Path

_config_path = Path(__file__).parent / "config.json"

with open(_config_path) as f:
    GUARDRAILS_CONFIG = json.load(f)

__all__ = ["GUARDRAILS_CONFIG"]
4. Create pe_policies/config.json

Use the same policy structure defined in PE_FIRM_POLICY above. Here's a condensed view:

{
  "version": 1,
  "pre_flight": {
    "version": 1,
    "guardrails": [
      { "name": "Contains PII", "config": { "entities": ["CREDIT_CARD", "EMAIL_ADDRESS", "US_SSN", "..." ], "block": true }},
      { "name": "Moderation", "config": { "categories": ["sexual", "hate", "violence", "..."] }}
    ]
  },
  "input": {
    "version": 1,
    "guardrails": [
      { "name": "Jailbreak", "config": { "confidence_threshold": 0.7, "model": "gpt-4.1-mini" }},
      { "name": "Off Topic Prompts", "config": { "confidence_threshold": 0.7, "model": "gpt-4.1-mini", "system_prompt_details": "..." }}
    ]
  },
  "output": {
    "version": 1,
    "guardrails": [
      { "name": "Contains PII", "config": { "entities": ["CREDIT_CARD", "EMAIL_ADDRESS", "..."], "block": true }}
    ]
  }
}

See PE_FIRM_POLICY in the Centralized Policy section for the full configuration with all entities and categories.

Note: The "block": true setting is required for the PII guardrail in the output stage. Without it, PII will be detected and masked but won't trigger a block.

5. Create pyproject.toml
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "pe-policies"
version = "0.1.0"
description = "PE Firm AI Agent Policy Configuration"
requires-python = ">=3.9"
dependencies = []

[tool.setuptools.packages.find]
include = ["pe_policies*"]

[tool.setuptools.package-data]
pe_policies = ["*.json"]
6. Push to GitHub
git add .
git commit -m "Initial policy package"
git remote add origin https://github.com/yourorg/pe-policies.git
git push -u origin main
7. Install and use from any project
pip install git+https://github.com/yourorg/pe-policies.git
from pe_policies import GUARDRAILS_CONFIG
from guardrails import GuardrailsOpenAI

client = GuardrailsOpenAI(config=GUARDRAILS_CONFIG)
### All calls now have governance automatically applied!

Putting It All Together

Here's the complete pattern for a governed agent system:


Improving & Optimizing

With the governed system running, we now evaluate, tune, and stress-test it.

Evaluating Your Guardrails

Building guardrails is only half the battle - you need to know they actually work. The OpenAI Guardrails library includes a built-in evaluation framework that measures precision, recall, and F1 scores against labeled test data.

Metric What It Measures Why It Matters
Precision Of all blocked queries, how many should have been blocked? High precision = few false positives (legitimate queries blocked)
Recall Of all bad queries, how many did we catch? High recall = few false negatives (threats getting through)
F1 Score Harmonic mean of precision and recall Balanced measure of overall performance

The trade-off: high precision with low recall means threats slip through; high recall with low precision blocks legitimate queries. Adjust confidence_threshold to find the right balance.

Step 1: Load the Test Dataset

The evaluation framework expects a JSONL file where each line contains:

  • id: Unique identifier for the test case
  • data: The input text (plain string or multi-turn JSON)
  • expected_triggers: Dict mapping each guardrail name to true/false

The full dataset (21 samples covering PII, moderation, jailbreak, off-topic, and mixed cases) is in eval_data/guardrail_test_data.jsonl. Below we load it and inspect the coverage.

Step 2: Create the Eval Config

We use PE_FIRM_POLICY directly as the eval config - evaluate what you deploy. This covers all three stages: pre-flight (PII, Moderation), input (Jailbreak, Off Topic), and output (PII).

Step 3: Run the Evaluation

You can run evals via CLI or programmatically. Here's both approaches:

Eval Best Practices

  1. Build diverse test sets: Include edge cases, adversarial examples, and legitimate queries
  2. Balance your dataset: Ensure roughly equal positive and negative examples per guardrail
  3. Run evals on policy changes: Before deploying updated confidence_threshold values
  4. Benchmark across models: Use --mode benchmark to compare gpt-5.2-mini vs gpt-5.2 for LLM-based guardrails
  5. Automate in CI/CD: Run evals on every policy repo change to catch regressions
### Benchmark mode compares models and generates ROC curves
guardrails-evals \
  --config-path config.json \
  --dataset-path test_data.jsonl \
  --mode benchmark \
  --models gpt-5.2-mini gpt-5.2

Automated Feedback Loop for Threshold Tuning

Manually tuning confidence_threshold values based on eval results is tedious. The Guardrail Feedback Loop automates this: it runs evals, analyzes precision/recall gaps, adjusts thresholds, re-validates, and saves the tuned config when metrics improve.

The loop includes oscillation prevention - if threshold adjustments keep flip-flopping, it reduces step size and eventually stops tuning that guardrail.

Step 1: Create a Tunable Configuration

We derive the tunable config directly from PE_FIRM_POLICY - the same config our GuardrailAgent uses - so we're tuning the actual production guardrails. The only change is overriding confidence_threshold values to an intentionally high starting point.

LLM-based guardrails like Jailbreak and Off Topic Prompts use confidence thresholds to decide when to trigger. The threshold controls the trade-off:

  • Higher threshold (e.g., 0.95): More conservative, fewer false positives, but may miss some threats
  • Lower threshold (e.g., 0.5): More sensitive, catches more threats, but may block legitimate queries

For this demo, we'll start with an intentionally high threshold (0.95) so you can see the tuner detect low recall and automatically decrease it.

Step 2: Create a Test Dataset

The feedback loop needs labeled test data to measure guardrail performance. Each test case specifies:

  • data: The text to evaluate - for conversation-aware guardrails (Jailbreak, Prompt Injection), we use multi-turn format with the agent's system prompt included so the guardrail evaluates in the same context as production
  • expected_triggers: Which guardrails should fire (true) or not (false)

Include both positive examples (should trigger) and negative examples (should not trigger) for precision/recall measurement. We also include adversarial PE-domain-camouflaged attacks and borderline-but-legitimate queries to stress-test false positive/negative rates.

Step 3: Run the Feedback Loop

Now we run the automated tuning process. The GuardrailFeedbackLoop will:

  1. Run an initial evaluation to get baseline metrics
  2. Compare precision/recall against our targets (90% each)
  3. Adjust thresholds based on which metric is underperforming
  4. Re-run evals to measure the impact
  5. Repeat until targets are met or max iterations reached

What to expect: With our intentionally high threshold (0.95), the initial eval will show low recall (the guardrail misses some jailbreak attempts). The tuner will detect this and decrease the threshold until recall meets the 90% target.

Watch the logs to see the loop's decision-making in action.

Step 4: Review the Results

After tuning completes, we can inspect what changes were made:

  • Threshold changes: How the confidence_threshold was adjusted
  • Metric improvements: Changes in precision, recall, and F1 score
  • Convergence status: Whether targets were achieved or tuning stopped early

The tuned configuration is automatically saved for use in production.

CLI Usage

You can also run the feedback loop from the command line:

### Basic usage
python tune_guardrails.py \
    --config eval_data/tunable_config.json \
    --dataset eval_data/input_guardrail_test_data.jsonl \
    --output tuning_results

### With custom targets
python tune_guardrails.py \
    --config eval_data/tunable_config.json \
    --dataset eval_data/input_guardrail_test_data.jsonl \
    --precision-target 0.95 \
    --recall-target 0.85 \
    --priority precision \
    --max-iterations 15 \
    --verbose

Output files include tuning_results/eval_config_tuned.json (optimized config), tuning_results/tuning_report_*.md (detailed report), and backups of original configs.


Red Teaming Your Guardrails with Promptfoo

Evals measured guardrail detection accuracy - "Did the guardrail fire correctly on known test cases?" But there's a harder question: "Can an attacker bypass your guardrails?"

Promptfoo is an open-source red teaming tool that auto-generates hundreds of adversarial inputs across 50+ vulnerability types - jailbreaks, prompt injections, PII extraction, off-topic hijacking, and more. Instead of writing test cases by hand, Promptfoo creates sophisticated, adaptive attacks and tests them against your actual application.

OpenAI Guardrails Eval Promptfoo Red Team
Tests guardrail detection accuracy (precision/recall) Tests whether adversarial inputs bypass guardrails
You write test cases manually Auto-generates hundreds of adversarial cases
Static dataset Adaptive attacks that evolve based on responses
"Did the guardrail fire?" "Can an attacker get through?"

Together they form a complete testing strategy: guardrails eval ensures detection quality, Promptfoo ensures resilience against real-world attacks.

How It Works Under the Hood

Promptfoo uses your existing OPENAI_API_KEY to power a three-phase process:

Your OPENAI_API_KEY
       │
       ▼
┌──────────────┐    adversarial     ┌──────────────────┐
│   Promptfoo   │─── prompts ──────▶│  Your target.py   │
│   (attacker)  │                   │  (GuardrailAgent) │
│   LLM generates◀── responses ────│                   │
│   & grades    │                   └──────────────────┘
└──────────────┘
       │
       ▼
  Red Team Report
  1. Generate: An LLM (defaults to gpt-5) generates adversarial prompts tailored to your application's purpose and selected plugins
  2. Attack: Each generated prompt is sent to your Python target script, which runs it through the governed agent (Runner.run)
  3. Grade: Another LLM call evaluates whether the response indicates a successful bypass or a proper block

Prerequisites and Cost

  • Promptfoo: Free, open source (MIT license)
  • Email verification: One-time free email check on first run (spam prevention, not a subscription)
  • LLM cost: Your standard OpenAI API usage for attack generation + grading. With numTests: 10 across ~9 plugins, expect ~100-200 API calls (a few dollars)
  • No subscription required -- your existing OPENAI_API_KEY is all you need

Step 1: Install Promptfoo

pip install promptfoo

Note: The pip package is a lightweight wrapper that requires Node.js 20+ installed on your system. Install Node via brew install node (macOS), sudo apt install nodejs npm (Ubuntu), or from nodejs.org.

Step 2: The Target Script

Promptfoo needs a way to talk to your governed agent. The file promptfoo/promptfoo_target.py bridges Promptfoo to your GuardrailAgent:

  • Receives each adversarial prompt from Promptfoo
  • Runs it through Runner.run(pe_concierge_governed, prompt) - the full agent with handoffs, tools, and centralized guardrails
  • Returns the response, or [BLOCKED] if any guardrail fires

The script recreates the same agent stack from the notebook: specialist agents, tools, custom pe_guardrail, PE_FIRM_POLICY, and the GuardrailAgent triage agent.

Step 3: The Red Team Config

The file promptfoo/promptfooconfig.yaml defines what to attack and how:

targets:
 - id: "python:promptfoo/promptfoo_target.py"
    label: "pe-concierge-governed"

purpose: >  # Application context improves attack quality
  A Private Equity firm front-desk AI assistant that handles deal screening,
  portfolio management, and investor relations...

redteam:
  numTests: 10  # Adversarial inputs per plugin
  plugins:                     # Generate adversarial inputs
   - hijacking                # Off-topic hijacking
   - pii:direct               # PII extraction attempts
   - prompt-extraction        # System prompt extraction
   - system-prompt-override   # Override system instructions
   - off-topic                # Off-topic manipulation
   - policy                   # Custom policy violations
  strategies:                  # Wrap inputs in evasion techniques
   - jailbreak                # Jailbreak wrapper patterns
   - prompt-injection         # Injection wrapper patterns
   - base64                   # Base64 encoding evasion
   - leetspeak                # l33tspeak encoding
   - rot13                    # ROT13 encoding evasion
   - crescendo                # Gradually escalating attacks

Plugins generate adversarial inputs targeting specific vulnerabilities. Strategies wrap those inputs in evasion techniques (jailbreak patterns, encoding, translation) to test whether guardrails can be bypassed beyond simple text matching. See the full plugin list for 131 available plugins.

Step 4: Run the Red Team

### Navigate to the promptfoo directory
cd promptfoo

### Generate adversarial inputs and run them against your agent
promptfoo redteam run

### View the interactive report
promptfoo redteam report

The report shows:

  • Pass/fail rate per vulnerability category
  • Severity levels for each finding
  • Concrete examples of inputs that bypassed guardrails
  • Suggested mitigations for each vulnerability

Sample Report

Here's what a successful red team report looks like -- 0 vulnerabilities across all categories, 33/33 tests defended:

The report breaks results into Risk Categories (Security & Access Control, Brand) and individual tests (Resource Hijacking, System Prompt Override, PII via Direct Exposure, Off-Topic Manipulation). Our GuardrailAgent with PE_FIRM_POLICY blocked 100% of the adversarial inputs.

Going Deeper

This demo used 5 plugins with numTests: 3 for a quick 33-probe scan. For production-grade assessments, increase the depth to 50+ probes per plugin and enable preset collections like owasp:llm (OWASP LLM Top 10), nist:ai:measure (NIST AI RMF), or mitre:atlas -- Promptfoo supports 131 plugins across security, compliance, trust & safety, and brand categories.

Interpreting Results

Any failures reveal gaps in your PE_FIRM_POLICY that need attention -- whether that's lowering thresholds, adding guardrails, or refining system prompts.

CI/CD Integration

Add red teaming to your deployment pipeline so guardrail changes are validated automatically:

### .github/workflows/redteam.yml
name: Red Team Guardrails
on:
  push:
    paths: ['guardrails/**']
jobs:
  redteam:
    runs-on: ubuntu-latest
    steps:
     - uses: actions/checkout@v4
     - uses: actions/setup-node@v4
        with: { node-version: 20 }
     - run: pip install promptfoo
     - run: promptfoo redteam run
     - run: promptfoo redteam report --output redteam-report.html
     - uses: actions/upload-artifact@v4
        with:
          name: redteam-report
          path: redteam-report.html

Key Takeaways

1. Governance enables adoption

By establishing clear guardrails upfront, you remove the fear and uncertainty that slows AI adoption. Teams can build confidently knowing policies are enforced automatically. Governance becomes an execution system that keeps adoption moving securely and at scale.

2. Use handoffs for specialization

Avoid one massive agent. Create specialists and let them collaborate. The handoff_description is key to good routing.

3. Layer your defenses

  • OpenAI Guardrails (client-level): Universal policies for all calls
  • Agents SDK guardrails (agent-level): Domain-specific validation

4. Trace everything (or nothing, for ZDR)

  • Use trace() to group operations for debugging
  • For ZDR compliance: disable tracing or use custom processors

5. Centralize policy, distribute capability

The policy-as-a-package pattern lets you:

  • Maintain governance in one place
  • Update policies without changing application code
  • Audit compliance across all projects

Next Steps

Initial Setup

  1. Create your policy repo using the template above
  2. Customize guardrails for your industry and compliance requirements
  3. Add custom trace processors if you need ZDR-compliant observability
  4. Document your policy alongside the code
  5. Set up CI/CD to test policy changes before deployment

Scaling AI Across Your Organization

When moving from prototype to production, consider how different user groups will interact with AI:

Role What They Build Governance Approach
Developers Custom agents, MCP connectors, integrations Safe defaults, reusable templates, evaluation pipelines
Power Users Configured assistants, automated workflows Pre-approved patterns, governed portals
End Users Content generation, data analysis Curated tools with embedded guardrails

This approach ensures everyone, from engineers to analysts, can leverage AI safely within appropriate boundaries.

Enabling Citizen Developers

Empower non-technical teams to build safely:

  • Provide templates for prompt packs, tool configurations, and evaluation checks
  • Create review lanes and publishing workflows that make it easy to build and deploy
  • Offer guardrailed sandboxes for experimentation without risking sensitive data
  • Establish clear promotion paths from prototype to production with governance checkpoints

Registries for Visibility

Treat AI assets as first-class governed resources by maintaining registries:

  • Agent Registry: Register all agents with owner, purpose, risk tier, and evaluation status
  • Tool Registry: Document MCP tools with authentication scopes, data access, and approval authority
  • Prompt Registry: Version and govern prompts like code, with lineage, rollback policies, and change controls

Registry metadata enables discoverability, auditing, and lifecycle management across your AI ecosystem.

Risk-Proportionate Controls

Not all AI use cases carry the same risk. Differentiate your controls:

  • Low-risk (internal productivity, non-sensitive data): Fast-track approval, minimal logging
  • Moderate-risk (customer-facing, operational data): Standard guardrails, audit trails
  • High-risk (PII, financial, regulated): Enhanced logging, human-in-the-loop, isolated environments

Apply proportionate controls, approvals, review, and detailed logging, only where necessary, keeping lightweight adoption fast and frictionless.

Preventing Shadow AI

Centralized governance helps prevent unauthorized AI tools from proliferating:

  • Make governed options easier than ungoverned alternatives
  • Provide clear adoption paths for different skill levels and use cases
  • Incorporate discovery mechanisms to detect and catalog unsanctioned AI activity
  • Offer support and training so teams don't go around the system

Early visibility allows governance teams to close gaps before they become systemic risks.

Standards Alignment

Align your governance practices with recognized frameworks:

  • NIST AI RMF - Risk management framework for AI systems
  • ISO/IEC 42001 - AI management system standard
  • Industry-specific requirements (HIPAA, SOX, GDPR, etc.)

Building on established standards creates external credibility alongside internal control.


Resources


FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.