Integrate OpenAI with Custom Ruby Providers

provider-ruby (Ruby Provider)

This example demonstrates how to create a custom Ruby provider for promptfoo that integrates with the OpenAI API.

You can run this example with:

npx promptfoo@latest init --example provider-ruby
cd provider-ruby

Overview

The Ruby provider allows you to use Ruby code as a provider in promptfoo evaluations. This example also demonstrates Ruby assertions for custom validation logic.

Ruby Provider is useful when you need to:

1. Call APIs from Ruby libraries
2. Implement custom logic before or after calling LLMs
3. Process responses in specific ways
4. Track token usage and other metrics

Ruby Assertions allow you to:

1. Write custom validation logic in Ruby
2. Access test context and variables
3. Return detailed grading results with scores and reasons
4. Reuse assertion logic across multiple tests

Environment Variables

This example requires the following environment variable:

• OPENAI_API_KEY - Your OpenAI API key

You can set this in a .env file or directly in your environment.

Requirements

• Ruby 2.7 or higher (with net/http and json from standard library)

Files

• provider.rb - The Ruby provider implementation that calls OpenAI's API
• assert.rb - Custom Ruby assertion functions for validation
• promptfooconfig.yaml - Configuration for promptfoo evaluation with proper YAML schema reference

Implementation Details

Ruby Provider (provider.rb)

The Ruby provider includes:

1. A call_api function that makes API calls to OpenAI
2. Token usage extraction from the API response
3. Multiple sample functions showing different ways to call the API

By default, the example is configured to use gpt-4.1-mini model, but you can modify it to use other models as needed.

Ruby Assertions

The example demonstrates three types of Ruby assertions:

1. Inline assertions - Simple one-line checks (e.g., output.length > 10)
2. Multiline assertions - Complex logic with detailed results and scores
3. External file assertions (assert.rb) - Reusable assertion functions

Ruby assertions can:

• Return boolean values for pass/fail
• Return numeric scores
• Return detailed GradingResult hashes with pass/fail, score, reason, and component results
• Access test context including variables, prompts, and provider responses

Expected Output

When you run this example, you'll see:

1. The prompts being submitted to your Ruby provider
2. Responses from the OpenAI API
3. Token usage statistics for each completion
4. Evaluation results in a table format

Run the example with:

npx promptfoo@latest evaluate -c examples/provider-ruby/promptfooconfig.yaml

Learn More

For more information, see the promptfoo documentation:

• Ruby Provider
• Ruby Assertions