What do I need to set up before using this prompt chain?

You need an OPENAI_API_KEY, Chromium browser installed via `npx playwright install chromium`, and a published ChatKit workflow ID from OpenAI's Agent Builder (e.g., `wf_68ffb8...`).

Can I use this with unpublished ChatKit workflows?

No, your workflow must be published in Agent Builder first — you need the workflow ID from the Publish step to configure the evaluation.

How do I set up multi-turn conversation testing?

Enable `stateful: true` in your config and run with `--max-concurrency 1`, then supply test variables for each conversation turn (e.g., an initial request followed by a follow-up question).

How do I get started with this example?

Run `npx promptfoo@latest init --example openai-chatkit`, then execute `npx promptfoo@latest eval` and `npx promptfoo@latest view` to test and visualize your ChatKit workflow.

Prompt Chain

Evaluate OpenAI ChatKit workflows with browser automation

Name: Openai Chatkit
Availability: OnlineOnly
Author: Promptfoo

A promptfoo example for evaluating OpenAI Agent Builder ChatKit workflows via browser automation.

Copy chain

Works with openaiplaywrightpromptfoo

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 21 days ago

Version 0.121.18

Add to Favorites

Why it matters

Test and validate OpenAI Agent Builder ChatKit workflows through automated browser-based evaluation, enabling quality assurance for conversational AI agents before deployment.

Outcomes

What it gets done

Drive browser automation to interact with published ChatKit workflows

Test multi-turn conversations with stateful session management

Simulate realistic user interactions across multiple conversation turns

Measure workflow performance and response quality across test cases

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-openai-chatkit | bash

Steps

Steps in the chain

Install Playwright browser

Get workflow ID from Agent Builder

Update promptfooconfig.yaml

Run the evaluation

View results

Overview

Openai Chatkit

A promptfoo example for evaluating OpenAI Agent Builder ChatKit workflows through browser automation, supporting stateful multi-turn conversations and simulated-user testing. Use it to evaluate a published Agent Builder ChatKit workflow. Requires Playwright Chromium, an OpenAI API key, and the workflow's published ID.

What it does

This promptfoo example evaluates ChatKit workflows built with OpenAI's Agent Builder, driving them through browser automation. It targets a specific published workflow by its workflow ID (for example, wf_68ffb83dbfc88190a38103c2bb9f421003f913035dbdb131), obtained from Agent Builder's Publish button.

When to use - and when NOT to

Use this example when you've built a conversational workflow in OpenAI's Agent Builder and want to evaluate it end to end, including multi-turn conversations that ask follow-up questions. It requires Playwright's Chromium browser installed, an OpenAI API key, and your workflow's published ID copied into the config - it evaluates a specific published Agent Builder workflow, not a general ChatKit integration.

Inputs and outputs

For workflows with follow-up questions, enable stateful mode and run with reduced concurrency so turns stay ordered:

providers:
  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID
    config:
      stateful: true

tests:
  - vars:
      message: 'I want to plan a party'
  - vars:
      message: 'For 20 people with a $500 budget'

Stateful mode should be run with --max-concurrency 1 to keep conversation turns in order; the default eval otherwise runs with --max-concurrency 4.

Integrations

For automated multi-turn testing, pair the stateful ChatKit provider with promptfoo's simulated-user provider, giving it natural-language instructions (for example, playing a user planning a birthday party) and a maxTurns limit, so the simulated user drives the conversation instead of hand-written per-turn test cases. See promptfoo's ChatKit and Simulated User provider docs, and OpenAI's own Agent Builder / ChatKit guide, for further configuration detail.

Who it's for

Teams building conversational workflows in OpenAI's Agent Builder who want automated, browser-driven evaluation of both single-turn and multi-turn behavior before shipping a workflow.

Source README

openai-chatkit (OpenAI ChatKit Workflow Evaluation)

Evaluate ChatKit workflows built with OpenAI's Agent Builder using browser automation.

Quick Start

npx promptfoo@latest init --example openai-chatkit
cd openai-chatkit

Prerequisites

Install Playwright browser:

npx playwright install chromium

Get your workflow ID from Agent Builder:
- Open platform.openai.com > Agent Builder
- Click Publish on your workflow
- Copy the workflow ID (e.g., wf_68ffb83dbfc88190a38103c2bb9f421003f913035dbdb131)
Update promptfooconfig.yaml with your workflow ID

Environment Variables

OPENAI_API_KEY - Your OpenAI API key

Run the Eval

npx promptfoo@latest eval --max-concurrency 4

View results:

npx promptfoo@latest view

Multi-Turn Conversations

For workflows that ask follow-up questions, enable stateful mode:

providers:
  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID
    config:
      stateful: true

tests:
  - vars:
      message: 'I want to plan a party'
  - vars:
      message: 'For 20 people with a $500 budget'

Run with --max-concurrency 1 for stateful mode.

Using with Simulated User

Test multi-turn conversations automatically:

providers:
  - id: openai:chatkit:wf_YOUR_WORKFLOW_ID
    config:
      stateful: true

defaultTest:
  provider:
    id: 'promptfoo:simulated-user'
    config:
      maxTurns: 5

tests:
  - vars:
      instructions: |
        You are planning a birthday party.
        Answer questions naturally and provide details when asked.