Test OpenAI Code Generation Responses

openai-responses (OpenAI Responses API Examples)

This directory contains examples for testing OpenAI's Responses API with promptfoo.

You can run this example with:

npx promptfoo@latest init --example openai-responses
cd openai-responses

Examples

Basic Responses API (promptfooconfig.yaml)

Basic example showing how to use the Responses API with GPT-5.6 Sol, Terra, and Luna, plus a GPT-4.1 comparison model.

External Response Format (promptfooconfig.external-format.yaml)

Example demonstrating how to load response_format configuration from external files. This is useful for:

• Reusing complex JSON schemas across multiple configurations
• Managing large schemas in separate files for better organization
• Version controlling schemas independently

This example compares inline vs. external file approach:

• Inline: JSON schema defined directly in the config
• External: JSON schema loaded from response_format.json using file:// syntax

Function Calling (promptfooconfig.function-call.yaml)

Example demonstrating function calling capabilities with the Responses API.

Function Callbacks (promptfooconfig.function-callback.yaml)

Example showing how to use function callbacks to execute functions locally instead of just returning the function call. This allows you to:

• Execute custom logic when the model calls a function
• Return the result directly to the test assertions
• Test end-to-end workflows including function execution

Key differences from regular function calling:

• Uses functionToolCallbacks to define JavaScript functions
• Functions are executed locally and results are returned
• Perfect for testing tool-using AI agents

Reasoning Models (promptfooconfig.reasoning.yaml)

Compare GPT-5.6 Sol, Terra, and Luna with GPT-6 Astra using explicit reasoning budgets.

GPT-5.1 (promptfooconfig.gpt-5.1.yaml)

Example demonstrating GPT-5.1's key features including:

• none reasoning mode: No reasoning tokens for fastest responses
• Verbosity control: Adjustable output length (low, medium, high)
• Reasoning effort levels: Compare none, medium, and high reasoning modes
• Coding tasks: Optimized for coding and problem-solving workflows

GPT-5.2 (promptfooconfig.gpt-5.2.yaml)

Example comparing GPT-5.2 with different reasoning effort levels:

• none: No reasoning tokens for fastest responses
• medium: Balanced reasoning for most tasks
• high: Maximum reasoning for complex problem-solving

GPT-5.5 (promptfooconfig.gpt-5.5.yaml)

Example comparing GPT-5.5 standard and pro models with different Responses API reasoning settings.

GPT-5.6 (promptfooconfig.gpt-5.6.yaml)

Example comparing the Sol, Terra, and Luna tiers. The gpt-5.6 alias routes to Sol. All tiers support max reasoning; Codex ultra is available for Sol and Terra rather than as a Responses API reasoning value.

GPT-6 Astra (promptfooconfig.gpt-6-astra.yaml)

Example using Astra with Responses, high reasoning, and structured output. Requires an OpenAI account with Astra access. Astra supports low, medium, high, xhigh, and max reasoning; tool calling requires Responses. See the provider documentation for pricing and hosting availability.

Image Processing (promptfooconfig.image.yaml)

Example demonstrating image input capabilities with vision models.

Web Search (promptfooconfig.web-search.yaml)

Example showing web search capabilities.

Prompt Caching (promptfooconfig.prompt-cache.yaml)

Example combining prompt_cache_key, prompt_cache_options, and included
web_search_call.results payloads in a Responses request.

Codex Models (promptfooconfig.codex.yaml)

Example using Codex models for code generation tasks.

MCP (Model Context Protocol) (promptfooconfig.mcp.yaml)

Example demonstrating OpenAI's MCP integration with remote MCP servers. This example uses the DeepWiki MCP server to query GitHub repositories.

MCP Features Demonstrated:

• Remote MCP server integration
• Tool filtering with allowed_tools
• Approval settings configuration
• Authentication headers (when needed)

Running the Examples

To run any of these examples:

### Basic Responses API example
npx promptfoo eval -c promptfooconfig.yaml

### External response format example
npx promptfoo eval -c promptfooconfig.external-format.yaml

### MCP example
npx promptfoo eval -c promptfooconfig.mcp.yaml

### Function calling example
npx promptfoo eval -c promptfooconfig.function-call.yaml

### Function callbacks example
npx promptfoo eval -c promptfooconfig.function-callback.yaml

### Reasoning models example
npx promptfoo eval -c promptfooconfig.reasoning.yaml

### GPT-5.1 example
npx promptfoo eval -c promptfooconfig.gpt-5.1.yaml

### GPT-5.2 example
npx promptfoo eval -c promptfooconfig.gpt-5.2.yaml

### GPT-5.5 example
npx promptfoo eval -c promptfooconfig.gpt-5.5.yaml

### GPT-5.6 example
npx promptfoo eval -c promptfooconfig.gpt-5.6.yaml

### GPT-6 Astra example
npx promptfoo eval -c promptfooconfig.gpt-6-astra.yaml --no-cache

### Prompt caching example
npx promptfoo eval -c promptfooconfig.prompt-cache.yaml

Prerequisites

• OpenAI API key set in OPENAI_API_KEY environment variable
• For MCP examples: Access to remote MCP servers (some may require authentication)

Notes

• The MCP example uses the public DeepWiki MCP server which doesn't require authentication
• For production use with MCP, carefully review the data being shared with third-party servers
• Some MCP servers may require API keys or authentication tokens in the headers configuration
• External file references support both JSON and YAML formats
• External files are resolved relative to the config file location