Test Replicate AI Models with Automated Evaluation

This is a minimal promptfoo example testing the Replicate provider, using Replicate's "hello-world" model - a simple, fast model well-suited for verifying a setup works. It verifies the API key is working, tests basic prompt templating, demonstrates simple assertions, and runs quickly (under 5 seconds per test).

Use this example as a first step to confirm a Replicate API key and promptfoo setup work correctly before moving to more advanced Replicate examples.

Requires a Replicate API token from replicate.com/account/api-tokens, set via:

export REPLICATE_API_TOKEN=r8_your_token_here

Run via npx promptfoo@latest init --example provider-replicate/quickstart then promptfoo eval. Once working, the example suggests swapping in other models by ID - replicate:meta/meta-llama-3-8b-instruct for text generation, or a versioned stability-ai/sdxl model ID for image generation - and points to three more advanced examples: provider-replicate/llama4-scout (Llama 4 text generation), provider-replicate/comprehensive (all provider features), and provider-replicate/image-generation (state-of-the-art image generation). Troubleshooting covers three issues: an API key error (verify REPLICATE_API_TOKEN is set correctly), a model-not-found error (check the model ID against replicate.com/explore), and a timeout (some models take 30-60 seconds on a first, cold-start run).

Built on the Replicate provider, demonstrated here with the hello-world model and pointing toward Llama 4 and Stable Diffusion XL models for more advanced use.

Engineers setting up the Replicate provider in promptfoo for the first time who want a fast, minimal test to confirm their API key and configuration work before running more complex text- or image-generation evaluations.