This promptfoo example evaluates replicate:meta/llama-4-scout-instruct - a 17B-parameter mixture-of-experts model with 16 experts - against meta-llama-3-8b-instruct as a reference point, both hosted on Replicate. Prompts are loaded from file://prompts.txt and rendered with a topic variable across seven subjects: AI/mixture-of-experts, multimodal AI, quantum computing, climate solutions, space exploration, healthy lifestyle tips, and creative writing.

Use this example as a template for benchmarking Llama 4 Scout's mixture-of-experts architecture against a smaller reference model like Llama 3 8B on general knowledge and creative tasks, mixing keyword checks with llm-rubric quality checks. It is a broad capability sweep rather than a benchmark focused on mixture-of-experts-specific behavior; one topic (space exploration) has no assertion at all, left for manual review.

Providers are configured with matching generation settings (temperature: 0.7, top_p: 0.9, ~1000 max tokens):

providers:
  # Llama 4 Scout - 17B parameter model with mixture-of-experts architecture
  - id: replicate:meta/llama-4-scout-instruct
    config:
      temperature: 0.7
      max_tokens: 1000
      top_p: 0.9

  # You can also compare with Llama 3 for reference
  - id: replicate:meta/meta-llama-3-8b-instruct
    label: llama-3-8b

Assertions mix contains-any keyword checks (e.g. quantum computing terms) with llm-rubric quality checks (e.g. "discusses both text and image understanding capabilities" for the multimodal AI topic).

Runs against Replicate-hosted meta/llama-4-scout-instruct and meta/meta-llama-3-8b-instruct, using promptfoo's contains-any and llm-rubric assertion types.

Teams evaluating Llama 4 Scout's mixture-of-experts model against a smaller Llama 3 baseline on Replicate across general knowledge and creative-writing tasks.