What are the main models available in this deep research setup?

The setup includes `o3-deep-research` (most powerful at $10/1M input, $40/1M output) and `o4-mini-deep-research` (faster and more affordable at $2/1M input, $8/1M output), plus their dated snapshots.

How long do deep research responses typically take?

Responses take 2-10 minutes to complete because the models perform extensive web searches and internal reasoning.

What configuration is required to use these deep research models?

You need an `OPENAI_API_KEY`, a generous `PROMPTFOO_EVAL_TIMEOUT_MS` (600000/10 minutes recommended), `max_output_tokens: 50000` or higher, and `tools: [{type: web_search_preview}]` in the provider config.

When should I not use deep research models?

Do not use them if you need fast, low-latency responses, as deep research models are built for depth rather than speed and consume significant tokens even before the visible answer.

Prompt Chain

Conduct Deep Web Research with AI-Powered Analysis

Name: Openai Deep Research
Availability: OnlineOnly
Author: Promptfoo

A promptfoo example for evaluating OpenAI's deep research models with web search, code, and MCP tools.

Copy chain

Works with openaipromptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 18 days ago

Version 0.121.18

Models

gpt 4o

Add to Favorites

Why it matters

Execute comprehensive research tasks that require extensive web searching, reasoning, and synthesis of current information into detailed reports with citations. The asset handles long-running research queries that demand multiple searches and deep analysis to produce thorough, up-to-date answers.

Outcomes

What it gets done

Automatically search the web multiple times to gather comprehensive information on complex topics

Synthesize search results and reasoning into detailed research reports with inline citations

Handle machine learning and technical research queries requiring current data

Generate structured responses with annotations, URLs, and citation details for verification

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-openai-deep-research | bash

Steps

Steps in the chain

Set OpenAI API key

Run evaluation with timeout

Prompt Enhancement - Clarification

Prompt Enhancement - Prompt rewriting

Prompt Enhancement - Deep research

Overview

Openai Deep Research

A promptfoo example for evaluating OpenAI's deep research models, which autonomously search the web and cite sources, via the Responses API with optional code interpreter and MCP tool integration. Use it to evaluate deep research model quality. Expect 2-10 minute response times, set max_output_tokens to 50,000+, and use an extended eval timeout.

What it does

This promptfoo example evaluates OpenAI's deep research models - which perform extensive web searches and reasoning through the Responses API - on machine learning and space exploration topics, checking that responses contain relevant technical terminology and handle web search results and citations correctly. Available models are o3-deep-research (most powerful, $10/1M input and $40/1M output tokens) and the faster, more affordable o4-mini-deep-research ($2/1M input, $8/1M output), each with a dated snapshot version too.

When to use - and when NOT to

Use this example when you need to evaluate a deep research model's ability to autonomously search the web and synthesize a cited answer. These models can take 2-10 minutes per task, use significant tokens for internal reasoning (max_output_tokens should be set to 50,000 or higher to avoid incomplete responses), and may require special OpenAI access - persistent 429 errors can indicate an access restriction rather than a simple rate limit.

Inputs and outputs

Set a long evaluation timeout before running, since a single task can take minutes:

export PROMPTFOO_EVAL_TIMEOUT_MS=600000
promptfoo eval

providers:
  - id: openai:responses:o4-mini-deep-research
    config:
      max_output_tokens: 50000
      tools:
        - type: web_search_preview

Responses include output_text (the final report with inline citations), annotations (citation URLs and titles), web_search_call (details of searches performed), and code_interpreter_call (any code analysis performed) - so a response is more than just plain text, it carries structured evidence of what the model searched and computed.

Integrations

For production use, background mode (background: true, with an optional webhook_url) avoids client-side timeouts on long-running research tasks. Adding a code_interpreter tool lets the model analyze data with code alongside web search, and an mcp tool (with server_label, server_url, and require_approval: never, which deep research requires) connects the model to private data sources via MCP servers. For better results, consider a prompt-enhancement pipeline: use a faster model to clarify and expand the user's query with specific requirements before passing the enhanced prompt to the research model itself.

Who it's for

Teams evaluating or building on OpenAI's deep research models for research-heavy, citation-backed tasks who need to validate search quality, citation accuracy, and tool integrations like code interpreter or private MCP data sources before shipping. Common failure modes worth checking for explicitly: timeouts (raise PROMPTFOO_EVAL_TIMEOUT_MS), incomplete responses (raise max_output_tokens), and tool-validation errors from a missing or misconfigured web_search_preview tool entry - the troubleshooting guidance treats these as the three most common issues encountered when running this example.

Source README

openai-deep-research (OpenAI Deep Research Models)

You can run this example with:

npx promptfoo@latest init --example openai-deep-research
cd openai-deep-research

This example demonstrates OpenAI's deep research models with web search capabilities via the Responses API.

Important Notes

⚠️ Response Times: Deep research models can take 2-10 minutes to complete research tasks as they perform extensive web searches and reasoning.

⚠️ Token Usage: These models use significant tokens for internal reasoning. Always set high max_output_tokens (50,000+) to avoid incomplete responses.

⚠️ Access: Deep research models may require special access from OpenAI. Check your API access if you encounter persistent 429 errors.

Setup

Set your OpenAI API key:

export OPENAI_API_KEY=your-key-here

Run the evaluation with appropriate timeout:

### Set a 10-minute timeout for deep research tasks
export PROMPTFOO_EVAL_TIMEOUT_MS=600000
promptfoo eval

For local development:

PROMPTFOO_EVAL_TIMEOUT_MS=600000 npm run local -- eval -c examples/openai-deep-research/promptfooconfig.yaml

What's happening?

This example:

Tests OpenAI's o4-mini-deep-research model with web search tools
Evaluates research capabilities on machine learning and space exploration topics
Uses the model's ability to automatically search the web for current information
Checks that responses contain relevant technical terminology
Demonstrates handling of web search results and citations

The model automatically decides when to use web search to provide comprehensive, up-to-date answers.

Configuration Details

providers:
  - id: openai:responses:o4-mini-deep-research
    config:
      max_output_tokens: 50000 # Required for complete research responses
      tools:
        - type: web_search_preview # Required for deep research models
      # Optional parameters:
      # max_tool_calls: 50 # Control number of searches (default: unlimited)
      # background: true # Use background mode for long-running tasks
      # store: true # Store the conversation for 30 days

Available Models

o3-deep-research - Most powerful deep research model ($10/1M input, $40/1M output)
o3-deep-research-2025-06-26 - Snapshot version
o4-mini-deep-research - Faster, more affordable ($2/1M input, $8/1M output)
o4-mini-deep-research-2025-06-26 - Snapshot version

Advanced Features

Background Mode (Recommended)

For production use, run deep research tasks in background mode to avoid timeouts:

providers:
  - id: openai:responses:o4-mini-deep-research
    config:
      background: true
      webhook_url: https://your-api.com/webhook # Optional: Get notified when complete

Using Code Interpreter

Deep research models can analyze data using code:

providers:
  - id: openai:responses:o4-mini-deep-research
    config:
      tools:
        - type: web_search_preview
        - type: code_interpreter
          container:
            type: auto

MCP Server Integration

Connect to private data sources using MCP servers:

providers:
  - id: openai:responses:o4-mini-deep-research
    config:
      tools:
        - type: web_search_preview
        - type: mcp
          server_label: mycompany_mcp
          server_url: https://mycompany.com/mcp
          require_approval: never # Required for deep research

Prompt Enhancement

For better results, consider preprocessing user queries:

Clarification: Use a faster model to gather context
Prompt rewriting: Expand the query with specific requirements
Deep research: Pass the enhanced prompt to the research model

See the OpenAI Deep Research Guide for detailed examples.

Response Format

Deep research responses include:

output_text: The final research report with inline citations
annotations: Citation details with URLs and titles
web_search_call: Details of searches performed
code_interpreter_call: Any code analysis performed

Troubleshooting

Timeouts: Increase PROMPTFOO_EVAL_TIMEOUT_MS if evaluations time out
Incomplete responses: Increase max_output_tokens to 50,000 or higher
429 errors: May indicate rate limits or access restrictions
Tool validation errors: Ensure web_search_preview is configured

Best Practices

Always use high token limits: Set max_output_tokens: 50000 or higher
Handle long response times: Use background mode or set high timeouts
Monitor costs: These models use significant tokens for reasoning
Validate citations: Check that returned URLs are accessible
Consider prompt enhancement: Preprocess queries for better results