What's the difference between errors and failures in promptfoo?

Errors occur when the provider itself fails (e.g., the custom provider throws an error), while failures occur when the provider responds but an assertion doesn't pass. This example demonstrates how promptfoo reports these differently.

How do I run this example?

Run `npx promptfoo@latest init --example eval-errors-vs-failures`, then `cd eval-errors-vs-failures`, and execute `promptfoo eval`.

Does this example use a real LLM provider?

No, it uses a synthetic custom provider built into the example that throws errors when the prompt contains the literal string `!ERROR!`.

Prompt Chain

Distinguish Test Errors from Assertion Failures

Name: Eval Errors Vs Failures
Availability: OnlineOnly
Author: Promptfoo

Example chain that shows how to differentiate between provider errors and failed assertions in LLM testing workflows.

Copy chain

Works with promptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 19 days ago

Version 0.121.18

Add to Favorites

Why it matters

Demonstrate and validate the difference between provider-level errors (system failures) and test-level failures (failed assertions) in LLM evaluation workflows, ensuring proper error handling and test result interpretation.

Outcomes

What it gets done

Trigger intentional provider errors using special prompt markers

Separate system-level failures from assertion-based test failures

Validate error handling behavior in evaluation pipelines

Demonstrate proper test failure classification patterns

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-errors-vs-failures | bash

Overview

Eval Errors Vs Failures

A promptfoo example that contains a custom provider which throws errors when prompts include `!ERROR!`, demonstrating the distinction between test errors (provider failures) and test failures (failed assertions). Use this example to see how promptfoo distinguishes between test errors and test failures. The custom provider throws errors when prompts contain `!ERROR!`, allowing you to observe the different failure types.

What it does

This promptfoo example demonstrates the distinction between test errors and test failures during evaluation. It contains a custom provider that throws errors when the prompt includes !ERROR!, showing how promptfoo distinguishes between test errors (provider failures) and test failures (failed assertions).

Inputs and outputs

You provide prompts to the custom provider. Test cases that contain the !ERROR! string will trigger error conditions from the provider.

How to use

Initialize and run the example with these commands:

npx promptfoo@latest init --example eval-errors-vs-failures
cd eval-errors-vs-failures

The example contains a custom provider that throws errors when the prompt includes !ERROR!.

Source README

eval-errors-vs-failures (Errors Vs Failures)

You can run this example with:

npx promptfoo@latest init --example eval-errors-vs-failures
cd eval-errors-vs-failures

Usage

This directory contains a custom provider that throws errors when the prompt includes !ERROR!, demonstrating how promptfoo distinguishes between test errors (provider failures) and test failures (failed assertions).

FAQ

Common questions

Discussion