Evaluate Code Generation with Claude

Runs Promptfoo evals against the Anthropic Messages API - including llm-rubric model-graded assertions - by reusing an existing local Claude Code session instead of creating a separate Anthropic Console API key. Setting apiKeyRequired: false in the provider config tells Promptfoo to skip its upfront API key check, load the Claude Code OAuth credential from the local session, authenticate Messages API requests with a Bearer token plus the claude-code-20250219,oauth-2025-04-20 beta headers, and prepend the required Claude Code identity system block ("You are Claude Code, Anthropic's official CLI for Claude.") before your own system prompt. Both the main provider and the llm-rubric grader point at anthropic:messages:claude-sonnet-4-6 with apiKeyRequired: false, so the entire eval runs on your Claude subscription rather than a separate API key.

This is meant for Claude Pro or Max subscribers who already use the claude CLI on the same machine and want to run evals without provisioning a separate Anthropic Console API key. Requests made this way are expected to count against your Claude subscription the same way calls from the claude CLI do - check Anthropic's documentation for current billing behavior before assuming it's free or unmetered. If ANTHROPIC_API_KEY is set in your environment, Promptfoo will prefer it over the OAuth credential, so unset it first if you specifically want the eval to run on your Claude Code session instead of a separate key.

npx promptfoo@latest init --example anthropic/claude-code-session
cd anthropic/claude-code-session

Prerequisite: install and log in to Claude Code with claude /login. Promptfoo reads the OAuth credential Claude Code stores in either the macOS keychain (Claude Code-credentials) or $HOME/.claude/.credentials.json. Run the eval with promptfoo eval and inspect results with promptfoo view. Configuration lives in promptfooconfig.yaml.

Integrates with the Anthropic Messages API via the anthropic:messages:claude-sonnet-4-6 provider string, authenticated through the local Claude Code CLI's OAuth session rather than a standalone API key, and reuses the same provider for its llm-rubric model-graded grader. Current Claude Code billing behavior for these requests is documented at docs.claude.com/en/docs/claude-code/overview.

Claude Pro or Max subscribers already running the claude CLI locally who want to evaluate prompts against the Anthropic Messages API - authenticated over the same Bearer-token OAuth flow the CLI itself uses - with model-graded rubric assertions and without setting up separate Console API billing.