Spark
Sign In
Prompt Chain

Compare Claude Model Reasoning

Promptfoo example testing Claude's thinking feature, comparing reasoning quality between the Anthropic API and AWS Bedrock on logic puzzles.

Copy chain
Works with anthropicaws
Promptfoo
Own this? Claim it
92
Spark score
out of 100
Updated last month
Version code-scan-action-0.1
Models
claude 3 5 sonnet
Add to Favorites

Why it matters

Understand and compare the step-by-step reasoning processes of different Claude AI models. Gain insights into how models like Sonnet and Haiku arrive at their conclusions before presenting a final answer.

Outcomes

What it gets done

01

Analyze Claude Sonnet 4's thinking process.

02

Analyze Claude Haiku 4.5's thinking process.

03

Compare reasoning outputs between models.

04

Extract key steps from model thought processes.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-claude-thinking | bash

Steps

Steps in the chain

01
Initialize claude-thinking example
02
Set up environment variables
03
Run evaluation
04
View results

Overview

Claude Thinking

This promptfoo example enables Claude's thinking feature and compares reasoning output quality between Claude Sonnet 4 on the Anthropic API and Claude Haiku 4.5 on AWS Bedrock. Use it when deciding whether to enable Claude's thinking feature for a task, or when comparing reasoning quality between the Anthropic API and AWS Bedrock.

What it does

This promptfoo example demonstrates Claude's "thinking" feature, which surfaces the model's step-by-step reasoning before its final answer, and compares thinking output quality between Claude Sonnet 4 (via the Anthropic API) and Claude Haiku 4.5 (via AWS Bedrock) on two increasingly complex logic and algebra puzzles.

When to use - and when NOT to

Use it when you want to enable and evaluate Claude's extended-thinking feature - comparing reasoning quality across Claude models or between the Anthropic API and AWS Bedrock, tuning the thinking token budget, or scoring reasoning output with an LLM-based rubric. Do not use it if you only need standard, non-thinking Claude responses, since enabling thinking requires extra config and a larger token budget than a normal call.

Inputs and outputs

Requires ANTHROPIC_API_KEY for the Anthropic API path, or AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (or aws configure) for the Bedrock path. Thinking is enabled with thinking: {type: 'enabled', budget_tokens: 4096} in the provider config, with max_tokens set higher than budget_tokens (e.g. 8192). Test cases are of increasing complexity: the "8 Balls Problem", a classic logic puzzle requiring careful reasoning, and the "Train Meeting Problem", a traditional algebra word problem - both chosen to showcase Claude's ability to break down complex problems and show detailed thinking steps. Output includes a "Thinking:" section showing the reasoning steps before the final answer:

Thinking: Let me solve this step by step...
1. First, I'll divide the 8 balls into three groups...
2. In the first weighing, I'll compare groups A and B...
3. Based on the result, I can determine...

Final answer: We need exactly 2 weighings to find the heavier ball.

Integrations

Runs via npx promptfoo@latest init --example claude-thinking, then promptfoo eval and promptfoo view. Evaluated against both the Anthropic API (Claude Sonnet 4) and AWS Bedrock (Claude Haiku 4.5) providers, with LLM-based evaluation rubrics used to assess reasoning quality.

Who it's for

Teams deciding whether to enable Claude's thinking feature for a task, or comparing reasoning quality and cost between the Anthropic API and AWS Bedrock before choosing a deployment path.

Source README

claude-thinking (Claude Thinking)

This example demonstrates Claude's "thinking" capability, which allows you to see the model's step-by-step reasoning process before it provides a final answer. The example compares thinking outputs from Claude Sonnet 4 (Anthropic API) and Claude Haiku 4.5 (AWS Bedrock).

You can run this example with:

npx promptfoo@latest init --example claude-thinking
cd claude-thinking

What This Example Demonstrates

  • Using Claude's thinking feature to reveal step-by-step reasoning
  • Comparing thinking output quality between different Claude models
  • Comparing Anthropic API vs AWS Bedrock providers
  • Configuring the thinking token budget
  • Using LLM-based evaluation rubrics to assess reasoning quality

Environment Variables

This example requires:

For Anthropic API

For AWS Bedrock

  • AWS_ACCESS_KEY_ID - Your AWS access key
  • AWS_SECRET_ACCESS_KEY - Your AWS secret key
  • Or configure credentials via the AWS CLI: aws configure

Running the Example

After setting up environment variables:

### From the example directory
promptfoo eval
promptfoo view

Test Cases

This example includes several test cases of increasing complexity:

  1. 8 Balls Problem - A classic logic puzzle requiring careful reasoning
  2. Train Meeting Problem - A traditional algebra word problem

These test cases are specifically designed to showcase Claude's ability to break down complex problems and show detailed thinking steps.

How Claude Thinking Works

The thinking feature is enabled by setting special parameters in the provider configuration:

thinking:
  type: 'enabled'
  budget_tokens: 4096 # Controls how many tokens are allocated for thinking
max_tokens: 8192 # Must be greater than budget_tokens

When enabled, Claude's response will include a "Thinking:" section that shows its reasoning process before the final answer:

Thinking: Let me solve this step by step...
1. First, I'll divide the 8 balls into three groups...
2. In the first weighing, I'll compare groups A and B...
3. Based on the result, I can determine...

Final answer: We need exactly 2 weighings to find the heavier ball.

Additional Resources

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.