Scout Web for Llama4 Insights
Promptfoo example running Meta Llama 4 Scout mixture-of-experts model via Replicate, compared against Llama 3.
Why it matters
Leverage Llama4 to autonomously scout the web for relevant information and generate concise summaries, aiding in research and content creation.
Outcomes
What it gets done
Search the web for specific topics.
Extract key information from search results.
Summarize findings into digestible insights.
Generate code snippets based on research.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-llama4-scout | bash Steps
Steps in the chain
Overview
Llama4 Scout
A Promptfoo example running Meta's 17B-parameter, 16-expert mixture-of-experts Llama 4 Scout model on Replicate across five task categories, compared against Llama 3. Use when evaluating Llama 4 Scout's reasoning or creative capabilities, or comparing it against Llama 3.
What it does
A Promptfoo example running Meta's Llama 4 Scout model via Replicate, a 17-billion-parameter, natively multimodal model using a 16-expert mixture-of-experts architecture, part of the broader Llama 4 collection. It tests the model on five task categories: AI and mixture-of-experts architecture, probing the model's self-awareness of its own design, multimodal AI understanding, quantum computing as a complex technical topic, climate solutions as practical problem-solving, and creative writing for narrative and storytelling. The example is built to compare Llama 4 Scout against Llama 3 to surface concrete improvements, and demonstrates the key Replicate model-configuration options for Llama 4: temperature from 0.0 deterministic to 1.0 very random, max_tokens as a generation length cap, and top_p as the nucleus-sampling threshold for token selection. Suggested extensions include testing Llama 4 Maverick, a 128-expert variant, when available, adding image-understanding tests once multimodal features are fully enabled, comparing against other state-of-the-art models, and exploring how the mixture-of-experts architecture specifically affects different task types.
When to use - and when NOT to
Use when evaluating Llama 4 Scout's reasoning, creative, or technical-explanation capabilities via Replicate, or when comparing it against Llama 3 or other models. Not the right example for testing Llama 4 Maverick directly, since it's referenced only as a future customization and not built into this example, or for multimodal image-understanding tests, since those features are noted as not yet fully enabled in this configuration.
Inputs and outputs
Input is a REPLICATE_API_TOKEN environment variable plus the model's temperature, max_tokens, and top_p parameters. Output, after running the eval and viewing results, is Llama 4 Scout's responses across the five test categories, comparable against Llama 3's responses to the same prompts.
npx promptfoo@latest init --example provider-replicate/llama4-scout
cd provider-replicate/llama4-scout
Integrations
Built on Replicate's API via Promptfoo's Replicate provider, targeting Meta's Llama 4 Scout model and, by extension, the broader Llama 4 collection including the larger Llama 4 Maverick variant.
Who it's for
Developers evaluating Llama 4 Scout's reasoning and creative capabilities, or comparing it against Llama 3 or other models, who want a ready-to-run Promptfoo example with the key Replicate configuration options already demonstrated.
Source README
provider-replicate/llama4-scout (Replicate Llama 4 Scout)
You can run this example with:
npx promptfoo@latest init --example provider-replicate/llama4-scout
cd provider-replicate/llama4-scout
This example demonstrates how to use Replicate to run the new Llama 4 Scout model, a cutting-edge 17 billion parameter model with 16 experts using mixture-of-experts architecture.
About Llama 4 Scout
Llama 4 Scout is part of the Llama 4 collection of natively multimodal AI models. Key features:
- 17 billion parameters with 16 experts
- Mixture-of-experts architecture for enhanced performance
- Natively multimodal - enables text and multimodal experiences
- Industry-leading performance in text and image understanding
Environment Variables
This example requires the following environment variable:
REPLICATE_API_TOKEN- Your Replicate API key (get one at https://replicate.com/account/api-tokens)
You can set this in a .env file or directly in your environment:
export REPLICATE_API_TOKEN=your_api_token_here
What This Example Does
This example:
- Tests the Llama 4 Scout model on various analytical and creative tasks
- Demonstrates the model's advanced reasoning capabilities
- Compares Llama 4 Scout with Llama 3 to show improvements
- Shows how to configure Replicate model parameters for optimal results
Running the Example
- Set your Replicate API token (see above)
- Run the evaluation:
promptfoo eval
- View the results:
promptfoo view
Model Configuration
The example demonstrates key Replicate configuration options for Llama 4:
temperature: Controls randomness (0.0 = deterministic, 1.0 = very random)max_tokens: Maximum number of tokens to generatetop_p: Nucleus sampling threshold for token selection
Test Cases
The example includes tests for:
- AI and mixture-of-experts architecture - Testing the model's self-awareness
- Multimodal AI - Exploring the model's understanding of multimodal capabilities
- Quantum computing - Complex technical topics
- Climate solutions - Practical problem-solving
- Creative writing - Narrative and storytelling abilities
Customizing the Example
You can modify this example to:
- Test Llama 4 Maverick (128 experts) when available
- Add image understanding tests (when multimodal features are enabled)
- Compare against other state-of-the-art models
- Explore the mixture-of-experts architecture's impact on different tasks
Notes
- Llama 4 Scout uses a mixture-of-experts approach for efficient computation
- The model excels at both analytical and creative tasks
- Response quality benefits from the 16-expert architecture
- Part of the Llama 4 ecosystem with multimodal capabilities
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.