Evaluate Claude Opus 4.8 on Advanced Coding Tasks

This promptfoo example exercises Opus-tier Claude models on hard coding tasks, comparing Claude Opus 5 against Claude Opus 4.8 at xhigh effort, plus Opus 5 at low effort so you can see the effort tradeoff on your own tasks. Because Opus 5 and Opus 4.8 share the same $5/$25 pricing, this is a like-for-like capability comparison rather than a cost comparison. The example evaluates three task types: bug diagnosis across multiple system boundaries, production-quality code generation with proper error handling, and code review with nuanced, prioritized feedback.

The two models behave differently and the example's config reflects that. Opus 5 runs adaptive thinking by default - an omitted thinking block does not mean "no thinking," so max_tokens (which caps thinking plus the final answer) needs real headroom; on this example's bug-diagnosis task, Opus 5 at xhigh spent about 4.4k tokens thinking, and an 8k budget truncated the answer mid-sentence, which is why this config uses 16k. Disabling thinking on Opus 5 (thinking: { type: disabled }) is only accepted at effort high or below - pairing it with xhigh or max returns a 400, and promptfoo drops the rejected disabled setting (keeping your effort level) with a one-time warning. effort is the main cost lever for Opus 5, ranging low through max; the source recommends starting at xhigh for coding and agentic work then sweeping downward, noting low and medium are unusually strong on Opus 5.

Opus 4.8, by contrast, builds on Opus 4.7 with the same feature set (no breaking API changes) and improved capability on complex reasoning and long-horizon agentic coding, but its adaptive thinking is opt-in: without an explicit thinking: { type: adaptive } block, Opus 4.8 runs without extended thinking even at high effort, so this example sets that explicitly on the 4.8 provider. Opus 4.8's effort defaults to high (setting it explicitly to high is a no-op), with xhigh also available. Both models reject temperature, top_p, and top_k at the model level; promptfoo omits these automatically so they shouldn't be set in config.

Use this example when you want a ready-made, runnable benchmark comparing Claude Opus 5 against Opus 4.8 on genuinely hard coding tasks, or when you want a working reference config for correctly setting thinking and effort on either model - since getting these wrong (e.g. an under-sized max_tokens on Opus 5, or forgetting to opt Opus 4.8 into adaptive thinking) silently degrades results without an obvious error. It's not a fit if you're evaluating non-coding tasks or lower-tier Claude models - this example is scoped specifically to Opus-tier coding and agentic work.

Input is your own ANTHROPIC_API_KEY (or equivalent credentials for Bedrock/Vertex/Azure) plus the example's bundled prompts and test cases. Set up and run:

npx promptfoo@latest init --example anthropic/opus-4-8-coding
cd opus-4-8-coding
export ANTHROPIC_API_KEY=your_api_key_here
npx promptfoo@latest eval
npx promptfoo@latest view

Output is a promptfoo evaluation run comparing Opus 5 (xhigh and low effort) against Opus 4.8 (xhigh effort) across the bug-diagnosis, code-generation, and code-review tasks, viewable in promptfoo's local results UI.

Beyond the native Anthropic Messages API, Opus 4.8 is also reachable through AWS Bedrock (bedrock:us.anthropic.claude-opus-4-8 or bedrock:converse:us.anthropic.claude-opus-4-8), Google Vertex (vertex:claude-opus-4-8 with config.region: global), and Azure AI Foundry (pointing anthropic:messages:claude-opus-4-8 at https://<resource>.services.ai.azure.com/anthropic via apiBaseUrl). Across all four providers, promptfoo automatically omits the unsupported sampling parameters for Opus 4.8; the native Anthropic provider logs a one-time warning if they're set explicitly, while the Bedrock, Vertex, and Azure paths omit them silently.

Developers and teams evaluating which Opus-tier Claude model and effort setting to use for coding or agentic workloads, and anyone building a promptfoo eval who needs a correct reference for configuring thinking and effort on Opus 5 versus Opus 4.8 across the Anthropic, Bedrock, Vertex, and Azure provider surfaces.