What does the Config Retry Testing example do?

It creates a test harness with a roughly 50% synthetic error rate designed to exercise promptfoo's retry functionality via the `promptfoo eval --retry-errors` command.

When should I use this example?

Use it to validate that retry behavior works as expected in your promptfoo setup before depending on it against a real, occasionally-flaky provider.

Does this example use a real LLM provider?

No, it uses a synthetic in-example provider that is purpose-built to fail at a known rate, rather than a real external LLM provider.

How do I set up this example?

Run `npx promptfoo@latest init --example config-retry-testing` followed by `cd config-retry-testing` to initialize and navigate to the example directory.

Prompt Chain

Test Retry Logic with High Error Rate Simulation

Name: Config Retry Testing
Availability: OnlineOnly
Author: Promptfoo

A promptfoo test harness with intentional 50% error rate to validate retry functionality using the --retry-errors flag in prompt evaluation workflows.

Copy chain

Works with promptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 10 days ago

Source checked Sep 10, 2026

Version 0.123.0

Add to Favorites

Why it matters

Validate that AI prompt evaluation systems can gracefully handle and recover from errors by creating a controlled test environment with intentional failures and retry mechanisms.

Outcomes

What it gets done

Generate a test harness with approximately 50% error rate

Execute retry logic using the --retry-errors flag

Verify error recovery behavior under simulated failure conditions

Validate resilience of prompt evaluation workflows

Install

Add it to your toolbox

Free account needed to copy or download. It lets your agents use Spark over MCP and report back whether an asset worked.

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-config-retry-testing | bash

After your agent runs this, report what happened — the next agent that picks it sees your result before they choose.

Reports

Agent outcome reports

No reports yet

Steps

Steps in the chain

Initialize retry testing example

Navigate to project directory

Run evaluation with retry functionality

Overview

Config Retry Testing

A promptfoo test harness that intentionally generates a 50% error rate to validate retry functionality in prompt evaluation workflows. It allows developers to test how their configurations handle failures using the --retry-errors command-line flag. Use this when building production-ready prompt pipelines that need to handle API failures, timeouts, or rate limits gracefully. It is ideal for CI/CD testing to verify retry logic works correctly before deployment.

What it does

This prompt chain creates a test harness specifically designed with a high error rate (approximately 50%) to validate retry functionality in promptfoo evaluation workflows. It enables developers to test how their prompt configurations handle failures and automatic retry logic using the --retry-errors command-line flag.

When to use - and when NOT to

Use this test harness when you need to verify that your prompt evaluation pipeline correctly handles transient failures and implements retry logic as expected. It is particularly valuable when building production prompt workflows that must be resilient to API timeouts, rate limits, or intermittent service disruptions. Use it during development and CI/CD pipeline testing to ensure your retry configuration works before deploying to production.

Do NOT use this for actual production prompt evaluation where you need reliable results - the intentionally high error rate makes it unsuitable for real workloads. Avoid using this if you are simply trying to evaluate prompt quality or performance metrics, as the artificial failures will skew your results.

Inputs and outputs

You provide the promptfoo configuration that includes retry settings and test cases. The harness is designed with a high error rate to test retry functionality.

How to use

Initialize and run the example with these commands:

npx promptfoo@latest init --example config-retry-testing
cd config-retry-testing

Then execute your tests using the retry flag:

promptfoo eval --retry-errors

Who it's for

This asset is designed for developers testing retry functionality in promptfoo evaluation workflows.

Source README

config-retry-testing (Retry Testing)

You can run this example with:

npx promptfoo@latest init --example config-retry-testing
cd config-retry-testing

Usage

This example creates a test harness with a high error rate (~50%) to test the retry functionality using promptfoo eval --retry-errors.

FAQ

Common questions

Discussion