Build and Compare Go-Based AI Providers for Testing

This example demonstrates how to structure and implement custom Go-based providers for promptfoo. It includes two complete implementations of the same provider interface, showing how to organize shared Go code in a single module, wrap OpenAI client functionality, and compare multiple provider implementations side-by-side in one configuration file.

Use this example when you need to build custom Go providers for promptfoo that integrate with OpenAI or other LLM services, when you want to compare different provider implementations using the same interface, or when you need a reference architecture for organizing Go-based provider code with shared utilities. Do not use this if you only need to call existing promptfoo providers without custom Go code, or if you require a language other than Go for your provider implementation.

You provide a Go environment (1.16 or later), an OpenAI API key, and the promptfoo configuration. Both provider implementations accept a prompt string and options map through the CallApi function interface.

The example integrates with the OpenAI Go client library (github.com/sashabaranov/go-openai v1.37.0) and promptfoo's evaluation framework. The directory structure supports shared code organization:

provider-golang/
├── go.mod            # Root module definition
├── main.go           # Root provider implementation
├── core/             # Supporting code
│   └── openai.go     # OpenAI client wrapper
├── pkg1/             # Shared utilities
│   └── utils.go      # Configuration
├── evaluation/       # Alternative implementation
│   └── main.go      # Provider with same interface
└── promptfooconfig.yml  # Config comparing both implementations

Both implementations use the same interface:

func CallApi(prompt string, options map[string]interface{}) (string, error)

This example serves developers building custom Go-based providers for promptfoo and those who need reference code for structuring Go provider implementations with shared utilities.

To get started:

npx promptfoo@latest init --example provider-golang
cd provider-golang

Install dependencies:

go get github.com/sashabaranov/go-openai@v1.37.0

Set your API key:

export OPENAI_API_KEY=your_key_here

Run the comparison:

px promptfoo eval

The configuration file compares both implementations with different settings:

providers:
  - id: 'file://evaluation/main.go:CallApi'
    label: 'Provider in evaluation/'

  - id: 'file://main.go:CallApi'
    label: 'Provider in root'
    config:
      reasoning_effort: 'high'