What models does this promptfoo example use?

This example uses `replicate:meta/llama-4-scout-instruct` for text generation tasks and `replicate:image:stability-ai/sdxl` (pinned by version hash) for image generation tasks.

How do I test image generation with Replicate in promptfoo?

Use the `replicate:image:` prefix followed by the model name and version hash in your config, then write assertions that check for markdown image references pointing to hosted URLs.

What types of assertions are used in this example?

Text generation tests use `contains-any` keyword checks and `javascript` assertions to validate structure (like haiku line count), while image generation tests check for markdown image references using provider-scoped assertions.

When should I use this example?

Use this as a reference template if you need to test multiple Replicate-hosted model types (both text and image) in a single promptfoo config and want working examples of assertion patterns for each modality.

Prompt Chain

Summarize Web Content Comprehensively

Name: Replicate Comprehensive
Availability: OnlineOnly
Author: promptfoo/promptfoo

A promptfoo example testing both text (Llama 4 Scout) and image (SDXL) generation through the Replicate provider.

Copy chain

promptfoo/promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Add to Favorites

Why it matters

Automate the process of gathering and synthesizing information from multiple web sources to provide a comprehensive summary.

Outcomes

What it gets done

Fetch content from various web pages.

Extract key information and data points.

Synthesize findings into a coherent summary.

Identify and present relevant details from search results.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-replicate-comprehensive | bash

Overview

Replicate Comprehensive

This promptfoo example tests both Replicate-hosted text generation (Llama 4 Scout) and image generation (SDXL) in one config, with modality-specific assertions. Use it as a reference for combining text and image generation tests against Replicate in one promptfoo config.

What it does

This promptfoo example exercises the Replicate provider for both text and image generation in a single config: replicate:meta/llama-4-scout-instruct for text tasks (haiku writing, quantum computing explanation) and replicate:image:stability-ai/sdxl:... for image generation (mountain landscape, abstract geometric art), each with its own assertion style.

When to use - and when NOT to

Use this example as a reference for testing multiple Replicate-hosted model types in one promptfoo config - text models via the standard provider syntax and image models via the replicate:image: prefix with a pinned model version hash. It is a working template combining both modalities' assertion patterns rather than a deep benchmark of either model's quality.

Inputs and outputs

Text generation tests combine keyword checks with structural JavaScript assertions targeted at a specific provider:

  - vars:
      task: 'Write a haiku about mountains'
    assert:
      - type: contains-any
        value: ['mountain', 'peak', 'high', 'snow', 'summit']
      - type: javascript
        value: |
          const lines = output.split('\n').filter(line => line.trim().length > 0);
          return {
            pass: lines.length >= 3,
            reason: `Found ${lines.length} lines, expected at least 3 for haiku`
          };
        provider: replicate:meta/llama-4-scout-instruct

Image generation tests instead check that the output is a markdown image reference pointing to a hosted URL: output.includes(', scoped to the SDXL image provider via the provider: field on the assertion.

Integrations

Runs against Replicate-hosted meta/llama-4-scout-instruct for text and stability-ai/sdxl (pinned by version hash) for images, using promptfoo's contains-any and per-assertion provider-scoped javascript checks.

Who it's for

Developers testing multiple Replicate model types - text and image generation - in one eval who want working examples of provider-scoped assertions for each modality.

Source README

yaml-language-server: $schema=https://promptfoo.dev/config-schema.json

description: Comprehensive Replicate provider testing

prompts:

'{{task}}'

providers:

Text generation with Llama 4

id: replicate:meta/llama-4-scout-instruct
config:
temperature: 0.5
max_new_tokens: 200

Image generation with SDXL

id: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
config:
width: 512
height: 512
num_inference_steps: 25

tests:

Text generation tests

vars:
task: 'Write a haiku about mountains'
assert:
- type: contains-any
  value: ['mountain', 'peak', 'high', 'snow', 'summit']
- type: javascript
  value: |
  // Basic check for haiku-like structure
  const lines = output.split('\n').filter(line => line.trim().length > 0);
  return {
  pass: lines.length >= 3,
  reason: Found ${lines.length} lines, expected at least 3 for haiku
  };
  provider: replicate:meta/llama-4-scout-instruct
vars:
task: 'Explain quantum computing in one paragraph'
assert:
- type: contains-any
  value: ['quantum', 'qubit', 'superposition', 'entanglement']
- type: javascript
  value: 'output.length > 100'
  provider: replicate:meta/llama-4-scout-instruct

Image generation test

vars:
task: 'A serene mountain landscape at sunset with a lake reflection'
assert:
- type: javascript
  value: |
  // Check that the output contains markdown image format
  return output && typeof output === 'string' && output.includes(';
  provider: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc
vars:
task: 'Abstract geometric art with vibrant colors and sharp angles'
assert:
- type: javascript
  value: |
  // Check that the output contains markdown image format
  return output && typeof output === 'string' && output.includes(';
  provider: replicate:image:stability-ai/sdxl:7762fd07cf82c948538e41f63f77d685e02b063e37e496e96eefd46c929f9bdc

FAQ

Common questions

Discussion