What AI models does this example use for text and image generation?

It uses Meta's Llama 3 8B Instruct for text generation (with Llama 3 70B as a larger alternative) and Stable Diffusion XL for image generation (with Playground v2.5 as an alternative). Both are accessed through the Replicate provider.

What are the approximate costs for using these models?

Llama 3 8B text generation costs roughly $0.0005 per 1K tokens, while Stable Diffusion XL image generation costs about $0.012 per image. Current pricing can be verified at replicate.com/pricing.

What assertion types are used in this example?

The example uses `contains-any` for keyword checks, `javascript` for custom validation logic, and `is-valid-openai-image-generation-response` to validate image URLs.

Prompt Chain

Evaluate and Refine Code with AI

Name: Comprehensive
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example testing Replicate's full range: Llama 3 text generation and Stable Diffusion XL image generation.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated today

Version 0.121.19

Models

llama 3

Add to Favorites

Why it matters

Leverage AI to comprehensively evaluate code quality, identify potential issues, and generate refined code suggestions. This asset aims to streamline the code review process and improve overall code integrity.

Outcomes

What it gets done

Analyze code for errors and vulnerabilities.

Summarize code functionality and potential improvements.

Generate optimized or corrected code snippets.

Facilitate code review and refinement cycles.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-comprehensive | bash

Steps

Steps in the chain

Set your Replicate API token

Run the evaluation

View the results

Overview

Comprehensive

A promptfoo example testing Replicate's full capabilities - Llama 3 text generation and Stable Diffusion XL image generation - with realistic assertion types for each. Use when evaluating Replicate as a provider for both text and image generation and you need realistic assertion and cost reference points.

What it does

This example demonstrates the full capabilities of the Replicate provider in promptfoo, including text generation and image generation. Text generation uses Meta's Llama 3 8B Instruct model with configurable temperature, token limits, and sampling parameters, tested on creative writing and explanatory tasks. Image generation uses Stable Diffusion XL at a custom 512x512 resolution for faster generation, tested on both photorealistic and artistic styles.

When to use - and when NOT to

Use this example when you want a single reference covering both text and image generation through Replicate, including realistic assertion types for each modality. It is not exhaustive of Replicate's full model catalog - it demonstrates the pattern with Llama 3 and SDXL, and points to larger/alternative models (Llama 3 70B, Playground v2.5) as customization options.

Inputs and outputs

Set REPLICATE_API_TOKEN (from replicate.com/account/api-tokens), then scaffold and run:

npx promptfoo@latest init --example provider-replicate/comprehensive
cd provider-replicate/comprehensive

Run with promptfoo eval and view with promptfoo view. Provider config sets temperature (e.g. 0.5 for balanced creativity) and max_new_tokens (e.g. 200 to limit response length) on replicate:meta/meta-llama-3-8b-instruct. Assertion types used: contains-any (keyword checks), javascript (custom validation logic), and is-valid-openai-image-generation-response (validates image URLs).

Integrations

Integrates the Replicate API as a promptfoo provider, covering replicate:meta/meta-llama-3-8b-instruct for text (with replicate:meta/meta-llama-3-70b-instruct as a larger alternative) and Stable Diffusion XL for images (with replicate:image:playgroundai/playground-v2.5-1024px-aesthetic as an alternative). Documented cost: roughly $0.0005 per 1K tokens for Llama 3 8B text generation and about $0.012 per image for SDXL, with current pricing checkable at replicate.com/pricing. Troubleshooting notes: Replicate has rate limits (add delays between tests if needed), large models can take 30-60 seconds with the provider handling polling automatically, and generated image URLs are temporary so should be saved if needed long-term.

Who it's for

Teams evaluating Replicate as a provider for both text and image generation who want a working reference covering realistic parameters, assertion types, and cost/rate-limit considerations for each modality before building their own eval.

Source README

provider-replicate/comprehensive (Comprehensive Replicate Testing)

You can run this example with:

npx promptfoo@latest init --example provider-replicate/comprehensive
cd provider-replicate/comprehensive

This example demonstrates the full capabilities of the Replicate provider in promptfoo, including text generation and image generation.

Environment Variables

This example requires:

REPLICATE_API_TOKEN - Your Replicate API key (get one at https://replicate.com/account/api-tokens)

Set this in your environment:

export REPLICATE_API_TOKEN=r8_your_api_token_here

Features Demonstrated

1. Text Generation with Llama 3

Uses Meta's Llama 3 8B Instruct model
Configurable temperature, token limits, and sampling parameters
Demonstrates creative writing and explanatory tasks

2. Image Generation with SDXL

Uses Stable Diffusion XL for high-quality image generation
Custom resolution settings (512x512 for faster generation)
Tests both photorealistic and artistic styles

Running the Example

Set your Replicate API token:

export REPLICATE_API_TOKEN=your_token_here

Run the evaluation:
```
promptfoo eval
```
View the results:
```
promptfoo view
```

Understanding the Configuration

Provider Configuration

providers:
  - id: replicate:meta/meta-llama-3-8b-instruct
    config:
      temperature: 0.5 # Balanced creativity
      max_new_tokens: 200 # Limit response length

Assertion Types

The example uses various assertion types:

contains-any: Checks for specific keywords
javascript: Custom validation logic
is-valid-openai-image-generation-response: Validates image URLs

Customization Ideas

Try Different Models:
- Text: replicate:meta/meta-llama-3-70b-instruct (larger, more capable)
- Images: replicate:image:playgroundai/playground-v2.5-1024px-aesthetic
Adjust Parameters:
- Increase temperature for more creative outputs
- Change image dimensions for different aspect ratios
- Modify num_inference_steps for quality vs speed tradeoff
Add More Tests:
- Code generation tasks
- Language translation
- Style transfer for images
- Different artistic styles

Troubleshooting

Rate Limits: Replicate has rate limits; add delays between tests if needed
Timeouts: Large models may take 30-60 seconds; the provider handles polling automatically
Image URLs: Generated images are temporary; save them if needed for long-term use

Cost Considerations

Text generation: ~$0.0005 per 1K tokens (Llama 3 8B)
Image generation: ~$0.012 per image (SDXL)
Check current pricing at https://replicate.com/pricing

FAQ

Common questions

Discussion