Integrate OpenAI with Custom Ruby Providers

This example demonstrates how to create a custom Ruby provider for promptfoo that integrates with the OpenAI API, and also demonstrates Ruby assertions for custom validation logic. The Ruby provider lets you use Ruby code as a provider in promptfoo evaluations - useful for calling APIs from Ruby libraries, implementing custom logic before or after calling LLMs, processing responses in specific ways, or tracking token usage and other metrics. Ruby assertions let you write custom validation logic in Ruby, access test context and variables, return detailed grading results with scores and reasons, and reuse assertion logic across multiple tests.

Use this example when you want to wrap an LLM call in custom Ruby logic, or write assertions in Ruby instead of JavaScript/Python for teams that already work primarily in Ruby. It is not a general Ruby tutorial - it documents the specific call_api provider interface and the three Ruby assertion styles promptfoo supports.

Requires Ruby 2.7 or higher (using net/http and json from the standard library) and OPENAI_API_KEY. Scaffold with:

npx promptfoo@latest init --example provider-ruby
cd provider-ruby

Running the example shows the prompts submitted to the Ruby provider, OpenAI's responses, per-completion token usage statistics, and evaluation results in table format. Run with npx promptfoo@latest evaluate -c examples/provider-ruby/promptfooconfig.yaml. Three files make up the example: provider.rb (the Ruby provider implementation calling OpenAI's API, defaulting to gpt-4.1-mini), assert.rb (custom Ruby assertion functions), and promptfooconfig.yaml (the eval config).

The provider.rb file implements a call_api function with token-usage extraction and multiple sample calling patterns, wired into OpenAI's API. Three Ruby assertion styles are demonstrated: inline assertions (simple one-line checks like output.length > 10), multiline assertions (complex logic returning detailed results and scores), and external file assertions (assert.rb, for reusable assertion functions). Ruby assertions can return a boolean pass/fail, a numeric score, or a detailed GradingResult hash with pass/fail, score, reason, and component results, with access to test context including variables, prompts, and provider responses.

Ruby-centric teams building a custom LLM provider or writing custom eval validation logic who want to stay in Ruby rather than switching to JavaScript or Python for either the provider implementation or the assertions. Having three distinct assertion styles available - a quick inline boolean check, a multiline block for logic that needs a real score and explanation, and an external file for logic reused across many tests - means a team can start with the simplest inline form for a first test and graduate to assert.rb only once the same validation logic is actually needed in more than one place, instead of committing to a separate file up front for even trivial checks.