Spark
Sign In
Prompt Chain

Load External Test Configurations via File References

Loads promptfoo's defaultTest config from an external file via file:// for reuse across multiple projects.

Copy chain
Works with openai
Promptfoo
Own this? Claim it
89
Spark score
out of 100
Updated 18 days ago
Version 0.121.18
Add to Favorites

Why it matters

Enable developers to externalize and reuse test configuration defaults across multiple prompt testing projects by loading shared YAML/JSON files instead of duplicating inline configuration.

Outcomes

What it gets done

01

Reference external defaultTest configuration files using file:// syntax

02

Share common test assertions and thresholds across multiple projects

03

Maintain centralized test standards in separate configuration files

04

Apply cost and latency thresholds from external defaults to all test cases

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-config-external-defaulttest | bash

Overview

Config External Defaulttest

Demonstrates loading promptfoo's defaultTest configuration from an external YAML/JSON file via file:// syntax, instead of defining it inline, for reuse across multiple projects. Use when sharing common test configurations across projects or keeping the main promptfoo config file clean.

What it does

This promptfoo example demonstrates loading defaultTest configuration from an external YAML or JSON file via the file:// syntax, instead of defining it inline in the main config. Traditional inline defaultTest embeds assertions and options directly (e.g. cost/latency thresholds and a provider) in promptfooconfig.yaml; the external approach replaces that block with a single reference:

### New external file approach
defaultTest: file://shared/defaultTest.yaml

The referenced file (shared/defaultTest.yaml) contains the same configuration that would normally be written inline, loaded automatically and applied to all test cases that don't override the defaults.

When to use - and when NOT to

Use this approach when sharing common test configurations across multiple projects, keeping the main config file cleaner, managing complex test configurations separately, or version-controlling shared test configurations independently of the main prompt config.

Inputs and outputs

Run via npx promptfoo@latest init --example config-external-defaulttest then npx promptfoo@latest eval. The example's structure separates promptfooconfig.yaml (main config) from shared/defaultTest.yaml (shared default test configuration). Advanced usage supports referencing defaultTest files from a parent directory (file://../shared-configs/defaultTest.yaml) or an absolute project-root path (file://configs/testing/defaultTest.yaml).

Integrations

Built on promptfoo's file:// reference mechanism, applicable to defaultTest the same way it applies to prompts and other config values. Four stated benefits: reusability across multiple prompt configurations, maintainability by updating test defaults in one place, organization by separating complex test config from the main file, and collaboration via a shared central location for team test standards.

Who it's for

Teams running promptfoo evaluations across multiple projects who want a single, version-controlled source of truth for default test assertions and options, rather than duplicating or drifting inline defaultTest blocks across separate config files.

Source README

config-external-defaulttest (External Default Test Configuration)

This example demonstrates how to use the file:// syntax to load defaultTest configuration from an external file.

You can run this example with:

npx promptfoo@latest init --example config-external-defaulttest
cd config-external-defaulttest

Overview

Instead of defining defaultTest inline in your main configuration file, you can reference an external YAML or JSON file. This is useful for:

  • Sharing common test configurations across multiple projects
  • Keeping your main config file cleaner and more focused
  • Managing complex test configurations separately
  • Version controlling shared test configurations independently

Structure

external-defaulttest/
├── promptfooconfig.yaml      # Main configuration file
├── shared/
│   └── defaultTest.yaml      # Shared default test configuration
└── README.md                 # This file

Usage

Inline defaultTest (traditional approach)

### Traditional inline approach
defaultTest:
  assert:
    - type: cost
      threshold: 0.01
    - type: latency
      threshold: 5000
  options:
    provider: openai:o4-mini

External defaultTest (new approach)

### New external file approach
defaultTest: file://shared/defaultTest.yaml

The external file (shared/defaultTest.yaml) contains the same configuration that would normally be inline.

Benefits

  1. Reusability: Share the same defaultTest configuration across multiple prompt configurations
  2. Maintainability: Update test defaults in one place
  3. Organization: Keep complex test configurations separate from your main config
  4. Collaboration: Teams can maintain shared test standards in a central location

Running the Example

cd examples/config-external-defaulttest
npx promptfoo@latest eval

The external defaultTest configuration will be loaded automatically and applied to all test cases that don't override the defaults.

Advanced Usage

You can also reference defaultTest files from parent directories or absolute paths:

### Reference from parent directory
defaultTest: file://../shared-configs/defaultTest.yaml

### Reference from project root
defaultTest: file://configs/testing/defaultTest.yaml

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.