What API keys do I need to use this prompt chain?

You need both `OPENAI_API_KEY` and `ANTHROPIC_API_KEY` to run the Claude vs GPT comparison.

How do I set up this comparison tool?

Run `npx promptfoo@latest init --example compare-claude-vs-gpt`, then edit `promptfooconfig.yaml` to define your prompts and test cases before running the evaluation.

What output does this tool produce?

Running `promptfoo eval` generates side-by-side results comparing Claude and GPT responses, which you can view with `promptfoo view`.

Do I need to provide my own test cases?

Yes, this is a bare-bones starting point — you must edit the config to define your own prompts and test cases for comparison.

Prompt Chain

Compare LLM Performance

Name: Compare Claude Vs Gpt
Availability: OnlineOnly
Author: Promptfoo

Promptfoo starting point for comparing Claude and GPT models side by side on your own test cases.

Copy chain

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

claude gpt 4o

Add to Favorites

Why it matters

Evaluate and compare the performance of different large language models (LLMs) like Claude and GPT on various tasks. This asset helps you understand their strengths and weaknesses for informed decision-making.

Outcomes

What it gets done

Run comparative tests between LLMs.

Analyze and summarize LLM outputs.

Identify differences in model responses.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-compare-claude-vs-gpt | bash

Steps

Steps in the chain

Initialize example project

Navigate to project directory

Set environment variables

Edit configuration

Run evaluation

View results

Overview

Compare Claude Vs Gpt

This is a minimal promptfoo starting point for comparing Claude and GPT model responses side by side on custom test cases. Use it as a bare-bones starting point for comparing Claude and GPT models; not a ready-made comparison task.

What it does

This example runs a promptfoo evaluation comparing Anthropic Claude and OpenAI GPT models side by side on the same test cases.

When to use - and when NOT to

Use it as a starting point for comparing Claude and GPT model responses on your own prompts and test cases. Do not use it expecting a pre-built comparison task - the source provides only the generic setup and run steps, and you edit the config to define your own comparison.

Inputs and outputs

Requires OPENAI_API_KEY and ANTHROPIC_API_KEY. Inputs are whatever you define in promptfooconfig.yaml; running promptfoo eval produces side-by-side results, viewable with promptfoo view.

Integrations

Set up via npx promptfoo@latest init --example compare-claude-vs-gpt, then edit promptfooconfig.yaml before running the evaluation.

Who it's for

Developers who want a bare-bones promptfoo starting point for comparing Claude and GPT model responses on their own test cases.

Source README

compare-claude-vs-gpt (Claude Vs Gpt)

You can run this example with:

npx promptfoo@latest init --example compare-claude-vs-gpt
cd compare-claude-vs-gpt

Usage

To get started, set your OPENAI_API_KEY and ANTHROPIC_API_KEY environment variables.

Next, edit promptfooconfig.yaml.

Then run:

promptfoo eval

Afterwards, you can view the results by running promptfoo view

FAQ

Common questions

Discussion