What environment setup is required to run this suite?

You need `OPENAI_API_KEY` set in your environment or `.env` file, and you run the suite from the repository root using `npm run local -- validate` or `npm run local -- eval` commands with the appropriate config file.

What are the main components that make up this eval suite?

The suite consists of four files: `promptfooconfig.yaml` (the eval suite itself), `generatePolicyEvalPrompt.cjs` (an executable prompt that runs redteam generate and emits normalized JSON), `cases/*.yaml` (the native redteam generation configs being tested), and `tests/policy-generation.yaml` (case metadata and promptfoo assertions).

Prompt Chain

Test Policy Plugin Code Generation

Name: Evals
Availability: OnlineOnly
Author: Promptfoo

Promptfoo internal suite testing its own PolicyPlugin redteam test generator across five generation modes.

Copy chain

Works with github

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

gpt 4o

Add to Favorites

Why it matters

This asset evaluates the Policy Plugin's test generation capabilities. It ensures the plugin's code generation for tests is robust and accurate.

Outcomes

What it gets done

Evaluate the Policy Plugin's test generation.

Verify the accuracy of generated test code.

Debug potential issues in the test generation process.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-evals | bash

Steps

Steps in the chain

Run validation

Run evaluation

Generate single comparison case

Overview

Evals

An internal promptfoo suite validating the PolicyPlugin redteam test generator across five native generation modes via promptfoo's real CLI generation path. Use when validating changes to promptfoo's own PolicyPlugin redteam test generator before shipping them.

What it does

This suite evaluates the PolicyPlugin test generator itself. It compares five native promptfoo redteam generate cases: normal single-input generation, policy text with explicit test-generation instructions, modifier-driven generation (Spanish output), multi-input generation with coordinated document/query attacks, and log-analysis generation with PromptBlock: output. The eval flow runs promptfoo redteam generate against each case config under cases/, normalizes the generated YAML into a stable JSON payload, and feeds that payload into promptfoo assertions and llm-rubric checks through an executable prompt - keeping the suite on promptfoo's real CLI generation path instead of a custom harness provider.

When to use - and when NOT to

Use this suite when validating that promptfoo's own PolicyPlugin redteam generator produces correct output across its supported generation modes - single-input, instruction-driven, modifier-driven, multi-input, and log-analysis. It is not a redteam suite for testing a third-party model or application - it tests promptfoo's own test-generation logic.

Inputs and outputs

Requires OPENAI_API_KEY in the environment or .env. From the repository root:

npm run local -- validate -c src/redteam/plugins/policy/evals/promptfooconfig.yaml
npm run local -- eval -c src/redteam/plugins/policy/evals/promptfooconfig.yaml --env-file .env --no-cache

A single comparison case can be generated directly, e.g. npm run local -- redteam generate -c src/redteam/plugins/policy/evals/cases/normal-single-input.yaml -o /tmp/policy-normal.yaml --force.

Integrations

Four files make up the suite: promptfooconfig.yaml (the eval suite), generatePolicyEvalPrompt.cjs (an executable prompt that runs redteam generate for one case and emits normalized JSON), cases/*.yaml (the native redteam generation configs being compared), and tests/policy-generation.yaml (case metadata and promptfoo assertions).

Who it's for

Promptfoo maintainers and contributors validating that the PolicyPlugin redteam test generator behaves correctly across its supported generation modes before shipping changes to it.

Source README

policy evals

This suite evaluates the PolicyPlugin test generator itself.

It compares five native promptfoo redteam generate cases:

normal single-input generation
policy text with explicit test-generation instructions
modifier-driven generation (Spanish output)
multi-input generation with coordinated document / query attacks
log-analysis generation with PromptBlock: output

The eval flow is:

run promptfoo redteam generate against each case config under cases/
normalize the generated YAML into a stable JSON payload
feed that payload into Promptfoo assertions and llm-rubric checks through an executable prompt

That keeps the suite on Promptfoo's real CLI generation path instead of using a custom harness provider.

Prerequisites

OPENAI_API_KEY available in your environment or in .env

Run

From the repository root:

npm run local -- validate -c src/redteam/plugins/policy/evals/promptfooconfig.yaml
npm run local -- eval -c src/redteam/plugins/policy/evals/promptfooconfig.yaml --env-file .env --no-cache

To generate any single comparison case directly:

npm run local -- redteam generate -c src/redteam/plugins/policy/evals/cases/normal-single-input.yaml -o /tmp/policy-normal.yaml --force

Files

promptfooconfig.yaml - eval suite
generatePolicyEvalPrompt.cjs - executable prompt that runs redteam generate for one case and emits normalized JSON
cases/*.yaml - native redteam generation configs being compared
tests/policy-generation.yaml - case metadata and Promptfoo assertions

FAQ

Common questions

Discussion