What does the Config Retry Testing example do?

It creates a test harness with a roughly 50% synthetic error rate designed to exercise promptfoo's retry functionality via the `promptfoo eval --retry-errors` command.

When should I use this example?

Use it to validate that retry behavior works as expected in your promptfoo setup before depending on it against a real, occasionally-flaky provider.

Does this example use a real LLM provider?

No, it uses a synthetic in-example provider that is purpose-built to fail at a known rate, rather than a real external LLM provider.

How do I set up this example?

Run `npx promptfoo@latest init --example config-retry-testing` followed by `cd config-retry-testing` to initialize and navigate to the example directory.

Prompt Chain

Test Retry Logic with High Error Rate Simulation

Name: Config Retry Testing
Availability: OnlineOnly
Author: Promptfoo

Simulate high error rates to test and validate retry functionality in AI prompt evaluation pipelines.

Copy chain

Works with promptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 18 days ago

Version 0.121.18

Add to Favorites

Why it matters

Validate that AI prompt evaluation systems can gracefully handle and recover from errors by creating a controlled test environment with intentional failures and retry mechanisms.

Outcomes

What it gets done

Generate a test harness with approximately 50% error rate

Execute retry logic using the --retry-errors flag

Verify error recovery behavior under simulated failure conditions

Validate resilience of prompt evaluation workflows

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-config-retry-testing | bash

Steps

Steps in the chain

Initialize retry testing example

Navigate to project directory

Run evaluation with retry functionality

Overview

Config Retry Testing

A promptfoo example configuration that creates a test harness with approximately 50% error rate to test retry functionality. When you need to test promptfoo's retry functionality using the --retry-errors flag under simulated error conditions.

What it does

Config Retry Testing is a promptfoo example that creates a test harness with an intentionally high error rate (approximately 50%) to test the retry functionality using promptfoo eval --retry-errors.

When to use - and when NOT to

Use this example when you want to test promptfoo's retry functionality under simulated error conditions.

Do not use this as a production configuration. The artificially high error rate is designed solely for testing retry mechanisms, not for actual prompt evaluation workloads.

Inputs and outputs

You initialize the example configuration, which includes a test harness that simulates failures at approximately 50% rate. When you run evaluations with the --retry-errors flag, you can observe how the retry mechanism behaves under these simulated error conditions.

How to use

Initialize the example and navigate to its directory:

npx promptfoo@latest init --example config-retry-testing
cd config-retry-testing

Once initialized, run evaluations with the retry flag to test the retry functionality against the simulated high error rate.

Who it's for

This example is for developers working with promptfoo who want to test retry functionality.

Source README

config-retry-testing (Retry Testing)

You can run this example with:

npx promptfoo@latest init --example config-retry-testing
cd config-retry-testing

Usage

This example creates a test harness with a high error rate (~50%) to test the retry functionality using promptfoo eval --retry-errors.

FAQ

Common questions

Discussion