Spark
Sign In
Prompt Chain

Evaluate Search Results with a Rubric

Verify LLM outputs against real-time web search results using promptfoo's search-rubric assertion.

Copy chain
Promptfoo
Own this? Claim it
92
Spark score
out of 100
Updated last month
Version 0.121.18
Models
claude 3 opusgpt 4o
Add to Favorites

Why it matters

Automate the evaluation of search engine results using a predefined rubric to ensure quality and relevance. This asset helps in systematically assessing the output of search queries.

Outcomes

What it gets done

01

Define and apply a structured rubric for evaluating search results.

02

Classify search results based on relevance and quality criteria.

03

Summarize findings from evaluated search results.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-eval-search-rubric | bash

Steps

Steps in the chain

01
LLM generates response
02
Extract claim to verify
03
Search for current information
04
Verify against web data

Overview

Eval Search Rubric

Uses promptfoo's search-rubric assertion to verify LLM claims against live web search results, with differing setup across Anthropic, OpenAI, and Perplexity providers. Use when an eval needs to verify time-sensitive or current-event claims that a static reference answer can't capture.

What it does

This example demonstrates the search-rubric assertion type, which verifies that LLM outputs contain accurate, current information by searching the web in real time - useful for current events, stock prices, weather, recent company information, or any time-sensitive data. The LLM generates a response, the assertion extracts the claim to verify, a provider with web search capability searches for current information, and the assertion passes or fails based on whether the output matches. Provider support differs: Anthropic Claude needs an explicit web_search_20250305 tool configuration (launched 2025, priced at $10 per 1,000 searches plus standard token costs); OpenAI needs a web_search_preview tool configuration and works with gpt-5.1, o4-mini, and other Responses API models; Perplexity has built-in web search with no extra configuration. Configuration takes a value (the search query to verify) and an optional threshold (minimum accuracy score, 0-1). Search-rubric assertions add roughly 2-5 seconds of latency each, so caching (npx promptfoo eval --cache) is recommended during development.

When to use - and when NOT to

Use this when your eval needs to verify time-sensitive or current-event claims that a static reference answer can't capture reliably - stock prices, news, weather, or anything that changes after the model's training cutoff.

Not needed for evaluating factual claims about stable, unchanging information - a static reference-answer assertion is cheaper and faster than the added web-search latency and cost.

Inputs and outputs

Input is promptfooconfig.yaml with search-rubric assertions and API keys for whichever provider supplies the web search (Anthropic, OpenAI, or Perplexity). Output is npx promptfoo eval, ideally run with --cache during development to avoid repeated live searches.

Integrations

Supports Anthropic Claude (web_search_20250305 tool), OpenAI (web_search_preview tool, gpt-5.1/o4-mini and other Responses API models), and Perplexity (built-in web search).

Who it's for

Teams evaluating LLM outputs that must reflect current, real-world facts rather than static or training-time knowledge.

npx promptfoo@latest init --example eval-search-rubric
cd eval-search-rubric
Source README

eval-search-rubric (Search Rubric)

You can run this example with:

npx promptfoo@latest init --example eval-search-rubric
cd eval-search-rubric

This example demonstrates how to use the search-rubric assertion type to verify that LLM outputs contain accurate, current information.

Overview

The search-rubric assertion allows you to verify facts by searching the web in real-time. This is particularly useful for:

  • Current events and news
  • Stock prices and financial data
  • Weather information
  • Recent company information
  • Any time-sensitive data

Running the Example

npx promptfoo eval

How It Works

  1. The LLM generates a response to your prompt
  2. The search-rubric assertion extracts the claim you want to verify
  3. A provider with web search capabilities searches for current information
  4. The assertion passes or fails based on whether the output matches current web data

Provider Support

Anthropic Claude

  • Web search capabilities via tool configuration (launched in 2025)
  • Requires explicit web_search_20250305 tool configuration
  • Pricing: $10 per 1,000 searches plus standard token costs

OpenAI

  • Requires web_search_preview tool configuration
  • Works with gpt-5.1, o4-mini, and other Responses API models

Perplexity

  • Built-in web search capabilities
  • No additional configuration needed

Configuration

assert:
  - type: search-rubric
    value: 'search query to verify'
    threshold: 0.8 # Optional: minimum accuracy score (0-1)

Notes

  • Search rubric assertions add latency (2-5 seconds per assertion)
  • Use caching during development: npx promptfoo eval --cache
  • Be specific with your search queries for better results

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.