Build and Compare Go-Based AI Providers for Testing

This example demonstrates how to structure Go-based providers for promptfoo. It shows two implementations of the same provider interface that share common code, enabling comparison of multiple Go provider implementations side-by-side in a single evaluation run.

Use this example when you need to build custom providers in Go for promptfoo, want to compare different provider implementations with shared utilities, or need to integrate Go-based logic into your evaluation workflow.

Do NOT use this if you only need simple prompt testing without custom provider logic, or if you're looking for a production-ready provider rather than a reference implementation to build upon.

You provide a Go implementation of the CallApi function with signature func CallApi(prompt string, options map[string]interface{}) (string, error), along with a promptfooconfig.yml that references your Go files.

You receive evaluation results comparing both provider implementations, viewable through the promptfoo web interface.

The example integrates with OpenAI through the github.com/sashabaranov/go-openai@v1.37.0 client library, wrapped in a custom client implementation. It works within the promptfoo evaluation framework, allowing Go providers to be compared alongside other provider types.

The directory structure supports modular Go code:

provider-golang/
├── go.mod            # Root module definition
├── main.go           # Root provider implementation
├── core/             # Supporting code
│   └── openai.go     # OpenAI client wrapper
├── pkg1/             # Shared utilities
│   └── utils.go      # Configuration
├── evaluation/       # Alternative implementation
│   └── main.go      # Provider with same interface
└── promptfooconfig.yml  # Config comparing both implementations

This example is for developers building custom providers in Go for promptfoo evaluations and those comparing different provider implementation strategies.

To get started:

npx promptfoo@latest init --example provider-golang
cd provider-golang

Set your OpenAI API key:

export OPENAI_API_KEY=your_key_here

Install the required Go dependency:

go get github.com/sashabaranov/go-openai@v1.37.0

Run the evaluation:

npx promptfoo eval

Then view results:

npx promptfoo view

The config demonstrates comparing implementations:

providers:
  - id: 'file://evaluation/main.go:CallApi'
    label: 'Provider in evaluation/'

  - id: 'file://main.go:CallApi'
    label: 'Provider in root'
    config:
      reasoning_effort: 'high'