Spark
Sign In
Prompt Chain

Test OpenAI Code Generation Responses

A promptfoo example collection covering OpenAI Responses API features - reasoning, MCP, caching, and more.

Copy chain
Works with openai
Promptfoo
Own this? Claim it
92
Spark score
out of 100
Updated 17 days ago
Version 0.121.18
Models
gpt 4o
Add to Favorites

Why it matters

Automate the testing of AI-generated code to ensure quality and correctness. This asset helps verify that code produced by models like OpenAI meets specified requirements and standards.

Outcomes

What it gets done

01

Generate code snippets using OpenAI.

02

Write and execute tests against generated code.

03

Review and debug code based on test results.

04

Integrate AI code generation into testing pipelines.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-openai-responses | bash

Overview

Openai Responses

A promptfoo example collection covering isolated OpenAI Responses API features, including reasoning effort levels, function callbacks, prompt caching, MCP integration, and version-specific model tiers. Use it as a reference config per feature - reasoning, function calling, caching, MCP - rather than a single end-to-end example. Requires an OpenAI API key and, for MCP, access to the target server.

What it does

This directory of promptfoo examples tests OpenAI's Responses API across a wide range of features, each in its own config file: basic usage with GPT-5.5, the GPT-5.4 family, and a GPT-4.1 comparison; external response_format schemas loaded via file:// syntax instead of defined inline; function calling and, separately, function callbacks that execute JavaScript functions locally via functionToolCallbacks and feed the results back into test assertions; reasoning models like o1 and o3; image input with vision models; web search; prompt caching combining prompt_cache_key, prompt_cache_retention, and web_search_call.results; Codex models for code generation; and remote MCP integration against the public DeepWiki server.

When to use - and when NOT to

Use this collection when you need a working reference config for a specific Responses API feature rather than building one from scratch - each example isolates one capability so you can copy just the piece you need. The version-specific examples cover real differences worth knowing: GPT-5.1 and GPT-5.2 compare none/medium/high reasoning effort levels, GPT-5.5 compares standard and pro models, and GPT-5.6 compares Sol, Terra, and Luna tiers, where the gpt-5.6 alias routes to Sol, all tiers support max reasoning, and Codex ultra is available only for Sol and Terra as a separate offering rather than as a Responses API reasoning value.

Inputs and outputs

Each example runs as its own eval:

npx promptfoo eval -c promptfooconfig.yaml
npx promptfoo eval -c promptfooconfig.function-callback.yaml
npx promptfoo eval -c promptfooconfig.gpt-5.6.yaml
npx promptfoo eval -c promptfooconfig.mcp.yaml

The MCP example demonstrates remote server integration, tool filtering via allowed_tools, approval settings, and authentication headers when a server needs them - the same DeepWiki server used requires no authentication itself.

Integrations

Requires OPENAI_API_KEY; MCP examples additionally need access to whichever remote MCP server is targeted, which may require its own authentication tokens in the headers configuration. External file references for response_format support both JSON and YAML, resolved relative to the config file's own location rather than the working directory. For production MCP use, carefully review what data is being shared with third-party MCP servers before enabling broad tool access.

Who it's for

Teams building on OpenAI's Responses API who want ready-made, isolated reference configs for reasoning effort tuning, function callbacks, prompt caching, MCP tools, or version-specific features like GPT-5.6's tiered models, instead of assembling each config from documentation alone. The external-format example is specifically useful once a JSON schema grows large enough that version-controlling it separately from the eval config, and reusing it across multiple configs, becomes worth the extra file.

Source README

openai-responses (OpenAI Responses API Examples)

This directory contains examples for testing OpenAI's Responses API with promptfoo.

You can run this example with:

npx promptfoo@latest init --example openai-responses
cd openai-responses

Examples

Basic Responses API (promptfooconfig.yaml)

Basic example showing how to use the Responses API with GPT-5.5, the GPT-5.4 family (gpt-5.4-mini, gpt-5.4-nano), and a GPT-4.1 comparison model.

External Response Format (promptfooconfig.external-format.yaml)

Example demonstrating how to load response_format configuration from external files. This is useful for:

  • Reusing complex JSON schemas across multiple configurations
  • Managing large schemas in separate files for better organization
  • Version controlling schemas independently

This example compares inline vs. external file approach:

  • Inline: JSON schema defined directly in the config
  • External: JSON schema loaded from response_format.json using file:// syntax

Function Calling (promptfooconfig.function-call.yaml)

Example demonstrating function calling capabilities with the Responses API.

Function Callbacks (promptfooconfig.function-callback.yaml)

Example showing how to use function callbacks to execute functions locally instead of just returning the function call. This allows you to:

  • Execute custom logic when the model calls a function
  • Return the result directly to the test assertions
  • Test end-to-end workflows including function execution

Key differences from regular function calling:

  • Uses functionToolCallbacks to define JavaScript functions
  • Functions are executed locally and results are returned
  • Perfect for testing tool-using AI agents

Reasoning Models (promptfooconfig.reasoning.yaml)

Example showing how to use reasoning models (o1, o3, etc.) with specific configurations.

GPT-5.1 (promptfooconfig.gpt-5.1.yaml)

Example demonstrating GPT-5.1's key features including:

  • none reasoning mode: No reasoning tokens for fastest responses
  • Verbosity control: Adjustable output length (low, medium, high)
  • Reasoning effort levels: Compare none, medium, and high reasoning modes
  • Coding tasks: Optimized for coding and problem-solving workflows

GPT-5.2 (promptfooconfig.gpt-5.2.yaml)

Example comparing GPT-5.2 with different reasoning effort levels:

  • none: No reasoning tokens for fastest responses
  • medium: Balanced reasoning for most tasks
  • high: Maximum reasoning for complex problem-solving

GPT-5.5 (promptfooconfig.gpt-5.5.yaml)

Example comparing GPT-5.5 standard and pro models with different Responses API reasoning settings.

GPT-5.6 (promptfooconfig.gpt-5.6.yaml)

Example comparing the Sol, Terra, and Luna tiers. The gpt-5.6 alias routes to Sol. All tiers support max reasoning; Codex ultra is available for Sol and Terra rather than as a Responses API reasoning value.

Image Processing (promptfooconfig.image.yaml)

Example demonstrating image input capabilities with vision models.

Web Search (promptfooconfig.web-search.yaml)

Example showing web search capabilities.

Prompt Caching (promptfooconfig.prompt-cache.yaml)

Example combining prompt_cache_key, prompt_cache_retention, and included
web_search_call.results payloads in a Responses request.

Codex Models (promptfooconfig.codex.yaml)

Example using Codex models for code generation tasks.

MCP (Model Context Protocol) (promptfooconfig.mcp.yaml)

Example demonstrating OpenAI's MCP integration with remote MCP servers. This example uses the DeepWiki MCP server to query GitHub repositories.

MCP Features Demonstrated:
  • Remote MCP server integration
  • Tool filtering with allowed_tools
  • Approval settings configuration
  • Authentication headers (when needed)

Running the Examples

To run any of these examples:

### Basic Responses API example
npx promptfoo eval -c promptfooconfig.yaml

### External response format example
npx promptfoo eval -c promptfooconfig.external-format.yaml

### MCP example
npx promptfoo eval -c promptfooconfig.mcp.yaml

### Function calling example
npx promptfoo eval -c promptfooconfig.function-call.yaml

### Function callbacks example
npx promptfoo eval -c promptfooconfig.function-callback.yaml

### Reasoning models example
npx promptfoo eval -c promptfooconfig.reasoning.yaml

### GPT-5.1 example
npx promptfoo eval -c promptfooconfig.gpt-5.1.yaml

### GPT-5.2 example
npx promptfoo eval -c promptfooconfig.gpt-5.2.yaml

### GPT-5.5 example
npx promptfoo eval -c promptfooconfig.gpt-5.5.yaml

### GPT-5.6 example
npx promptfoo eval -c promptfooconfig.gpt-5.6.yaml

### Prompt caching example
npx promptfoo eval -c promptfooconfig.prompt-cache.yaml

Prerequisites

  • OpenAI API key set in OPENAI_API_KEY environment variable
  • For MCP examples: Access to remote MCP servers (some may require authentication)

Notes

  • The MCP example uses the public DeepWiki MCP server which doesn't require authentication
  • For production use with MCP, carefully review the data being shared with third-party servers
  • Some MCP servers may require API keys or authentication tokens in the headers configuration
  • External file references support both JSON and YAML formats
  • External files are resolved relative to the config file location

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.