What does this Promptfoo example evaluate?

This example evaluates Claude Opus 4.6's coding and reasoning capabilities on complex software engineering tasks, including code analysis across multi-file codebases, bug diagnosis in multi-system scenarios, handling ambiguous requirements, tradeoff reasoning, and production-ready code generation.

How do I run this evaluation?

Set up your `ANTHROPIC_API_KEY` as an environment variable, then run `npx promptfoo@latest eval` to execute the evaluation and `npx promptfoo@latest view` to see the results.

When should I use this example?

Use this when evaluating Claude Opus 4.6 specifically on complex, multi-file software engineering tasks that require judgment under ambiguity, rather than straightforward code generation from a clear specification.

What are the key capabilities this example tests?

The example tests bug diagnosis across multiple system boundaries, choosing appropriate data structures with reasoning, writing production-quality code with error handling, and analyzing architectural decisions to propose improvements.

Prompt Chain

Generate and Review Complex Code

Name: Opus 4 6 Coding
Availability: OnlineOnly
Author: Promptfoo

A Promptfoo example evaluating Claude Opus 4.6's coding, bug-diagnosis, and tradeoff-reasoning skills with a 128K extended-thinking budget.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 12 days ago

Version 0.121.19

Models

claudeclaude 3 opus

Add to Favorites

Why it matters

Leverage state-of-the-art coding and reasoning to tackle complex software engineering tasks, including handling ambiguity and performing tradeoff analysis.

Outcomes

What it gets done

Generate high-quality code for complex problems.

Analyze and review code for quality and correctness.

Debug and resolve issues in software development.

Evaluate tradeoffs in software design and implementation.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-opus-4-6-coding | bash

Steps

Steps in the chain

Set your API key

Run the evaluation

View results

Overview

Opus 4 6 Coding

A Promptfoo example evaluating Claude Opus 4.6's coding capabilities: multi-file code analysis, bug diagnosis, ambiguity handling, tradeoff reasoning, and production-quality code generation. Use it to evaluate or benchmark Claude Opus 4.6 on coding tasks requiring multi-file understanding and tradeoff reasoning, not simple single-file generation.

What it does

Runs a Promptfoo evaluation suite demonstrating Claude Opus 4.6's coding and reasoning capabilities on complex software engineering tasks involving ambiguity and tradeoff analysis. The evaluation covers five capability areas: complex code analysis across multi-file codebases and architectural decisions, bug diagnosis across multi-system scenarios, handling ambiguous requirements without hand-holding, reasoning through tradeoffs between different approaches, and generating production-ready code. The example highlights Opus 4.6's stated strengths - the source describes it as achieving the highest score on SWE-bench Verified among frontier models, and supporting extended-thinking budgets up to 128K tokens for complex reasoning.

When to use - and when NOT to

Use it to evaluate or benchmark Claude Opus 4.6 on coding tasks that require multi-file understanding, ambiguity handling, and tradeoff reasoning, rather than simple single-file code generation that wouldn't exercise these capabilities.

Inputs and outputs

Input is the Anthropic API key set as ANTHROPIC_API_KEY. Output is a Promptfoo evaluation run assessing bug diagnosis across system boundaries, data-structure choices with stated reasoning, production-quality code with error handling, and architectural analysis with proposed improvements - viewable via npx promptfoo@latest view.

Integrations

Built on Promptfoo (npx promptfoo@latest init/eval/view) and the Anthropic API via the Promptfoo Anthropic provider, targeting the Claude Opus 4.6 model specifically.

Who it's for

For developers evaluating or benchmarking Claude Opus 4.6's coding capabilities, or building their own Promptfoo evaluation suites modeled on this one, who need a working example that exercises multi-file reasoning, ambiguity handling, and tradeoff analysis rather than toy prompts.

### Set your API key
export ANTHROPIC_API_KEY=your_api_key_here

### Run the evaluation
npx promptfoo@latest eval

### View results
npx promptfoo@latest view

Source README

anthropic/opus-4-6-coding (Claude Opus 4.6 Advanced Coding)

This example demonstrates Claude Opus 4.6's state-of-the-art coding and reasoning capabilities, showcasing its ability to handle complex software engineering tasks with ambiguity and tradeoff analysis.

You can run this example with:

npx promptfoo@latest init --example anthropic/opus-4-6-coding
cd anthropic/opus-4-6-coding

What This Tests

Claude Opus 4.6 is the best model in the world for coding, agents, and computer use. This example evaluates:

Complex code analysis: Understanding multi-file codebases and architectural decisions
Bug diagnosis: Identifying root causes in complex, multi-system scenarios
Ambiguity handling: Making informed decisions when requirements are unclear
Tradeoff reasoning: Evaluating different approaches and explaining pros/cons
Code generation: Writing high-quality, production-ready code

Features Demonstrated

State-of-the-art coding: Opus 4.6 achieves the highest score on SWE-bench Verified among frontier models
Reasoning about tradeoffs: The model excels at analyzing different approaches and making informed decisions
Handling ambiguity: Unlike models that require hand-holding, Opus 4.6 figures things out
Extended thinking: Support for thinking budgets up to 128K tokens for complex reasoning

Running the Example

### Set your API key
export ANTHROPIC_API_KEY=your_api_key_here

### Run the evaluation
npx promptfoo@latest eval

### View results
npx promptfoo@latest view

Expected Results

The evaluation tests Opus 4.6's ability to:

Diagnose bugs across multiple system boundaries
Choose appropriate data structures with clear reasoning
Write production-quality code with proper error handling
Analyze architectural decisions and propose improvements

Learn More

FAQ

Common questions

Discussion