What types of contracts are included in the test cases?

The six test cases cover: a real-estate purchase agreement, a residential lease, a software development services agreement, a dispute-resolution clause, a marketing consulting agreement, and a product sale agreement.

How are the model outputs graded?

Each test case uses an `llm-rubric` assertion that checks whether the model's summary covers specific required facts, and the real-estate test case additionally includes a `javascript` check to ensure the output stays under 500 characters.

Is this tool suitable for production contract review?

No, this is an evaluation harness for comparing model outputs against hand-written expected facts, not a production contract-review tool. The six contracts are illustrative test fixtures rather than a comprehensive legal test suite.

Prompt Chain

Benchmark Cohere Prompts

Name: Cohere Benchmark
Availability: OnlineOnly
Author: Promptfoo

Promptfoo eval benchmarking Cohere, GPT-5, and Claude Sonnet on extracting and plain-English-explaining key contract clauses across six contract types.

Copy chain

Works with coherepromptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

claude 3 5 sonnetgpt 4ouniversal

Add to Favorites

Why it matters

Automate the evaluation and benchmarking of Cohere prompts to ensure optimal performance and identify areas for improvement.

Outcomes

What it gets done

Define and execute benchmark tests for Cohere prompts.

Analyze and summarize benchmark results.

Integrate with promptfoo for automated testing.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-cohere-benchmark | bash

Overview

Cohere Benchmark

A promptfoo eval config comparing Cohere, GPT-5, and Claude Sonnet on extracting and plain-English-explaining the three most important clauses from six different contract types, graded with fact-coverage rubrics and a length check. Use as a template for multi-provider contract-summarization benchmarking or as a promptfoo example combining llm-rubric grading with a deterministic javascript length assertion. Not a production contract-review tool.

What it does

A promptfoo evaluation config that benchmarks three LLM providers - cohere:command-a-03-2025, openai:gpt-5, and anthropic:claude-sonnet-4-6, all run at temperature 0 - on a single fixed task: extract the three most important clauses from a contract and explain each one in plain English. The same prompt template is applied across six distinct contract test cases, each with its own grading rubric.

The six contracts cover different domains and clause types: a real-estate purchase agreement (graded on capturing the $500,000 purchase price, the June 30, 2023 closing deadline, financing and appraisal contingencies, and the clear-title requirement); a residential lease (graded on the $2,000 monthly rent, July 1, 2023 start date and 12-month term, tenant utility responsibility, and written subletting approval requirement); a software development services agreement (graded on the August 15, 2023 start date and 18-month duration, $150,000 total paid in $8,333.33 monthly installments, 30-day termination notice, and three-phase delivery with milestones in Exhibit A); a dispute-resolution clause (graded on the mediation-before-litigation requirement, prevailing-party attorney fee recovery, and New York governing law); a marketing consulting agreement (graded on the $25,000 non-refundable fee payable on signing, 45-day marketing-plan delivery, and up-to-two-revisions right); and a product sale agreement (graded on delivery of 100 units of Product X by September 30, 2023, the $50/unit price totaling $5,000, 30-day payment terms, and a 5% late-delivery discount).

Each test case is graded with an llm-rubric assertion listing the specific facts the model's summary must cover, and the real-estate test case additionally asserts via a javascript check that the output stays under 500 characters, testing conciseness alongside factual coverage.

prompts:
  - |
    Extract the 3 most important clauses from this contract,
    and explain each one in plain English:

    {{contract}}

When to use - and when NOT to

Use this config as a template for benchmarking how different LLM providers perform on contract clause extraction and plain-English explanation, or as a starting point for building a broader legal-document-summarization eval suite. It's a good reference for combining llm-rubric fact-coverage grading with a hard javascript length constraint in the same test case. It is not itself a production contract-review tool - it is an evaluation harness comparing model outputs against hand-written expected facts, and the six contracts are illustrative test fixtures rather than a comprehensive legal test suite.

Inputs and outputs

Input is a {{contract}} variable populated per test case with contract text. Output is each provider's plain-English three-clause summary, scored against an llm-rubric rubric of required facts (and, for one case, a character-length assertion).

Integrations

Runs on promptfoo's YAML eval framework, calling three provider APIs directly by ID (cohere:command-a-03-2025, openai:gpt-5, anthropic:claude-sonnet-4-6) with llm-rubric grading (itself backed by an LLM judge) and a javascript assertion for deterministic length checking.

Who it's for

Teams evaluating which LLM provider best extracts and explains contract clauses, or engineers building a promptfoo eval suite who want a working example of multi-provider comparison with mixed rubric-based and deterministic assertions.

Source README

yaml-language-server: $schema=https://promptfoo.dev/config-schema.json

description: 'Cohere benchmark'

providers:

id: cohere:command-a-03-2025
config:
temperature: 0
id: openai:gpt-5
config:
temperature: 0
id: anthropic:claude-sonnet-4-6
config:
temperature: 0

prompts:

Extract the 3 most important clauses from this contract,
and explain each one in plain English:

{{contract}}

tests:

vars:
contract: |
Seller agrees to convey the property located at 123 Main St
to Buyer for a total purchase price of $500,000. Closing to
occur on or before June 30, 2023. Sale is contingent upon
Buyer obtaining financing and the property appraising for
at least the purchase price. Seller to provide a clear
title free of any liens or encumbrances...
assert:
- type: llm-rubric
  value: |
  The summary should cover:
  - The purchase price of $500,000
  - The closing deadline of June 30, 2023
  - The financing and appraisal contingencies
  - Seller's responsibility to provide clear title
- type: javascript
  value: output.length < 500
vars:
contract: |
The tenant agrees to pay a monthly rent of $2,000 for the apartment at 456 Elm Street, starting from July 1, 2023. The lease term is 12 months. The tenant is responsible for utilities. The landlord must approve any subletting in writing.
assert:
- type: llm-rubric
  value: |
  Summary should include:
  - Monthly rent amount of $2,000
  - Lease start date of July 1, 2023, and term of 12 months
  - Tenant's responsibility for utilities
  - Requirement for written approval for subletting
vars:
contract: |
This agreement between ABC Corp and XYZ Inc. for software development services shall commence on August 15, 2023, and continue for a period of 18 months. ABC Corp will pay XYZ Inc. a total of $150,000, payable in monthly installments of $8,333.33. Either party may terminate the agreement with 30 days' written notice. XYZ Inc. agrees to deliver the software product in three phases, with specific milestones outlined in Exhibit A.
assert:
- type: llm-rubric
  value: |
  Summary should cover:
  - The agreement's start date of August 15, 2023, and duration of 18 months
  - Payment terms of $150,000 total, in monthly installments of $8,333.33
  - Termination clause requiring 30 days' written notice
  - Delivery of software in three phases with milestones in Exhibit A
vars:
contract: |
In the event of a dispute arising under this contract, both parties agree to first seek resolution through mediation before pursuing any legal action. The prevailing party in any litigation will be entitled to recover reasonable attorney fees and costs. This contract is governed by the laws of the State of New York.
assert:
- type: llm-rubric
  value: |
  Summary should highlight:
  - Agreement to seek mediation before legal action
  - Prevailing party's right to recover attorney fees
  - Governing law of the State of New York
vars:
contract: |
The consultant agrees to provide marketing services to the client for a project fee of $25,000. The fee is non-refundable and payable upon signing of this agreement. The consultant will deliver a comprehensive marketing plan within 45 days of the agreement date. The client has the right to request up to two revisions of the marketing plan.
assert:
- type: llm-rubric
  value: |
  Summary should include:
  - Project fee of $25,000, non-refundable and payable upon signing
  - Delivery of a marketing plan within 45 days
  - Client's right to request up to two revisions
vars:
contract: |
The seller agrees to deliver 100 units of Product X to the buyer by no later than September 30, 2023. The buyer agrees to pay $50 per unit, for a total purchase price of $5,000. Payment is due within 30 days of delivery. If the seller fails to deliver the products on time, the buyer is entitled to a 5% discount on the total purchase price.
assert:
- type: llm-rubric
  value: |
  Summary should cover:
  - Delivery of 100 units of Product X by September 30, 2023
  - Purchase price of $50 per unit, totaling $5,000
  - Payment due within 30 days of delivery
  - 5% discount entitlement for late delivery

FAQ

Common questions

Discussion