Test AI Providers with OpenClaw

provider-openclaw (OpenClaw Agent Testing)

This example demonstrates how to use promptfoo with OpenClaw, a personal AI assistant framework.

Prerequisites

1. Install OpenClaw: npm install -g openclaw@latest
2. Run the onboarding wizard: openclaw onboard
3. Enable the HTTP API in ~/.openclaw/openclaw.json if you want Chat or Responses.
   These HTTP endpoints are disabled by default upstream:

   {
     "gateway": {
       "http": {
         "endpoints": {
           "chatCompletions": {
             "enabled": true
           },
           "responses": {
             "enabled": true
           }
         }
       }
     }
   }
   

4. Start the gateway: openclaw gateway (or restart if already running: openclaw gateway restart)

Setup

npx promptfoo@latest init --example provider-openclaw
cd provider-openclaw

Usage

npx promptfoo@latest eval

The provider auto-detects the gateway URL and bearer auth secret from the active OpenClaw config
(OPENCLAW_CONFIG_PATH when set, otherwise ~/.openclaw/openclaw.json). This includes local
bind/port, OPENCLAW_GATEWAY_PORT local port overrides, gateway.tls.enabled, and
gateway.mode=remote with gateway.remote.url.

Configuration

You can override auto-detection with explicit config:

providers:
  - id: openclaw:main
    config:
      gateway_url: http://127.0.0.1:18789
      auth_token: your-token-here
      # Use auth_password instead when gateway.auth.mode=password
      # Optional backend model override, sent as x-openclaw-model:
      backend_model: openai/gpt-5.6-terra

Or use environment variables:

export OPENCLAW_CONFIG_PATH=~/.openclaw/openclaw.json  # optional
export OPENCLAW_GATEWAY_URL=http://127.0.0.1:18789
### Or override only the local auto-detected port:
### export OPENCLAW_GATEWAY_PORT=18789
export OPENCLAW_GATEWAY_TOKEN=your-token-here
### Or, if your gateway uses password auth:
### export OPENCLAW_GATEWAY_PASSWORD=your-password-here

For openclaw:agent:*, promptfoo generates an isolated session key per call by default so evals do
not reuse your persistent OpenClaw main session. Set session_key explicitly if you want session
continuity. The WS provider signs OpenClaw's connect.challenge, persists issued device tokens,
and retries once with a cached device token when the gateway recommends it.

Use openclaw:embedding:main or openclaw:embeddings:<agent-id> for /v1/embeddings. Set
backend_model when you want a specific embedding model such as openai/text-embedding-3-small.

For password-mode gateways, use auth_password or OPENCLAW_GATEWAY_PASSWORD.