What does the providers filter do in promptfoo?

The providers filter lets you run specific tests against only certain providers instead of creating a cross-product of every test with every provider. It supports exact label matching, exact provider IDs, wildcards (e.g., `openai:*`), and prefix matching (e.g., `openai`).

How much does using providers filters reduce test cases?

In the example, filtering 3 tests against 2 providers produces exactly 3 test cases instead of the full 6-case cross-product—the quick-facts test runs only on the fast model, the complex-reasoning test runs only on the smart model, and the basic test runs on both.

Can I set a default provider filter for all tests?

Yes, you can use the `defaultTest.providers` setting to apply a filter globally to every test, and individual tests can override it for their specific case.

When should I use provider filtering?

Use it when different test cases are meant for different providers—such as a latency-sensitive test only relevant to a fast model or a reasoning test only relevant to a capable model—to avoid running every test against every provider.

Prompt Chain

Run Isolated Test Suites Per AI Provider

Name: Config Tests Per Provider
Availability: OnlineOnly
Author: Promptfoo

Runs isolated test suites per provider in promptfoo via the providers filter, avoiding the default test x provider cross-product.

Copy chain

Works with promptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 19 days ago

Version 0.121.18

Add to Favorites

Why it matters

Configure and execute separate test suites for different AI providers without cross-pollination, ensuring each provider is evaluated only against relevant test cases rather than running every test against every model.

Outcomes

What it gets done

Filter which providers run specific test cases using labels, exact IDs, or wildcards

Set default provider filters globally to apply across all tests

Match providers using patterns like exact IDs, wildcards, or prefix matching

Reduce test case count by running targeted evaluations instead of full cross-products

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-config-tests-per-provider | bash

Steps

Steps in the chain

Setup the example project

Understand the problem with cross-product testing

Configure provider filtering on test cases

Use matching patterns for provider selection

Set default provider filters globally

Overview

Config Tests Per Provider

Demonstrates running isolated per-provider test suites in promptfoo via the providers filter and defaultTest.providers, avoiding the default N-tests-times-M-providers cross-product. Use when different tests need to run against different providers without cross-pollination in a promptfoo evaluation.

What it does

This promptfoo example demonstrates running different tests against different providers in a single evaluation using the providers filter on test cases, avoiding the default cross-product behavior where N tests times M providers creates N*M test cases. Each test case can restrict which providers it runs against:

tests:
  - vars:
      question: 'Quick math question'
    providers:
      - fast-model # Only runs on fast-model
    assert:
      - type: latency
        threshold: 2000

  - vars:
      question: 'Complex reasoning'
    providers:
      - smart-model # Only runs on smart-model

The providers filter supports four matching patterns: an exact label match (fast-model matching a provider with label: fast-model), an exact provider ID (openai:gpt-4), a wildcard (openai:* matching all providers with that prefix), and a bare prefix (openai, equivalent to the wildcard form). A defaultTest.providers block applies a filter to all tests by default, which individual tests can override with their own providers list.

When to use - and when NOT to

Use this example when different tests need to run against different providers without cross-pollination - for example a fast/cheap model for simple checks and a stronger model for complex reasoning - rather than running every test against every configured provider.

Inputs and outputs

Run via npx promptfoo@latest init --example config-tests-per-provider then npx promptfoo@latest eval. In the worked example, instead of 6 test cases from 3 tests times 2 providers, provider filtering produces exactly 3: a quick-facts test running on fast-model only, a complex-reasoning test running on smart-model only, and a basic test running on both providers.

Integrations

Built on promptfoo's test-case providers filter and defaultTest.providers global default, working alongside any configured provider set (illustrated with OpenAI and Anthropic-style provider IDs).

Who it's for

Engineers running promptfoo evaluations across multiple providers who need isolated test suites per provider - fast models for cheap checks, stronger models for complex reasoning - without the combinatorial blowup of testing every case against every provider.

Source README

config-tests-per-provider (Isolated Test Suites)

This example demonstrates how to run different tests against different providers in a single evaluation using the providers filter on test cases.

What You'll Learn

How to filter which providers run specific tests
How to run separate test suites without cross-pollination
Using wildcards and labels for provider matching
Using defaultTest to set provider filters globally

Setup

npx promptfoo@latest init --example config-tests-per-provider
npx promptfoo@latest eval

The Problem

Without filtering, promptfoo creates a cross-product of all tests × providers. If you have 2 tests and 2 providers, you get 4 test cases. This example shows how to run specific tests only against specific providers.

Features Demonstrated

Provider Filtering

Each test case can specify which providers it should run against:

tests:
  - vars:
      question: 'Quick math question'
    providers:
      - fast-model # Only runs on fast-model
    assert:
      - type: latency
        threshold: 2000

  - vars:
      question: 'Complex reasoning'
    providers:
      - smart-model # Only runs on smart-model

Matching Patterns

The providers filter supports multiple matching patterns:

Pattern	Example	Matches
Label	`fast-model`	Provider with `label: fast-model`
Exact ID	`openai:gpt-4`	Provider with that exact ID
Wildcard	`openai:*`	All providers starting with `openai:`
Prefix	`openai`	All providers starting with `openai:`

Default Filters

Use defaultTest.providers to apply filters to all tests:

defaultTest:
  providers:
    - openai:* # All tests default to OpenAI only

tests:
  - vars:
      question: 'Question 1'
  - vars:
      question: 'Question 2'
    providers:
      - anthropic:* # Override for this test only

Result

Instead of 6 test cases (3 tests × 2 providers), this example runs exactly 3 test cases:

Quick facts test → fast-model only
Complex reasoning test → smart-model only
Basic test → both providers

Learn More

FAQ

Common questions

Discussion