Evaluate Web Search Quality with Custom Datasets

Evaluating Web Search Quality with a Custom Dataset

This notebook demonstrates how to evaluate a model's ability to retrieve correct answers from the web using the OpenAI Evals framework with a custom in-memory dataset.

Goals:

• Show how to set up and run an evaluation for web search quality.
• Provide a template for evaluating information retrieval capabilities of LLMs.

Environment Setup

We begin by importing the required libraries and configuring the OpenAI client.
This ensures we have access to the OpenAI API and all necessary utilities for evaluation.

Define the Custom Evaluation Dataset

We define a small, in-memory dataset of question-answer pairs for web search evaluation.
Each item contains a query (the user's search prompt) and an answer (the expected ground truth).

Tip:
You can modify or extend this dataset to suit your own use case or test broader search scenarios.

Define Grading Logic

To evaluate the model’s answers, we use an LLM-based pass/fail grader:

• Pass/Fail Grader:
  An LLM-based grader that checks if the model’s answer (from web search) matches the expected answer (ground truth) or contains the correct information.

Best Practice:
Using an LLM-based grader provides flexibility for evaluating open-ended or fuzzy responses.

Define the Evaluation Configuration

We now configure the evaluation using the OpenAI Evals framework.

This step specifies:

• The evaluation name and dataset.
• The schema for each item (what fields are present in each Q&A pair).
• The grader(s) to use (LLM-based pass/fail).
• The passing criteria and labels.

Best Practice:
Clearly defining your evaluation schema and grading logic up front ensures reproducibility and transparency.

Run the Model and Poll for Completion

We now run the evaluation for the selected models (gpt-4.1 and gpt-4.1-mini).

After launching the evaluation run, we poll until it is complete (either completed or failed).

Best Practice:
Polling with a delay avoids excessive API calls and ensures efficient resource usage.

Display and Interpret Model Outputs

Finally, we display the outputs from the model for manual inspection and further analysis.

• Each answer is printed for each query in the dataset.
• You can compare the outputs to the expected answers to assess quality, relevance, and correctness.

You can visualize the results in the evals dashboard by going to https://platform.openai.com/evaluations as shown in the image below:

In this notebook, we demonstrated a workflow for evaluating the web search capabilities of language models using the OpenAI Evals framework.

Key points covered:

• Defined a focused, custom dataset for web search evaluation.
• Configured an LLM-based grader for robust assessment.
• Ran a reproducible evaluation with the latest OpenAI models and web search tool.
• Retrieved and displayed model outputs for inspection.

Next steps and suggestions:

• Expand the dataset: Add more diverse and challenging queries to better assess model capabilities.
• Analyze results: Summarize pass/fail rates, visualize performance, or perform error analysis to identify strengths and weaknesses.
• Experiment with models/tools: Try additional models, adjust tool configurations, or test on other types of information retrieval tasks.
• Automate reporting: Generate summary tables or plots for easier sharing and decision-making.

For more information, see the OpenAI Evals documentation.