What is OrcaRouter and how does it work with this prompt chain?

OrcaRouter is an OpenAI-compatible adaptive routing gateway that evaluates prompts against multiple upstream models through a single endpoint. This example demonstrates calling specific upstream models like `openai/gpt-4o-mini` directly, or using the `orcarouter:orcarouter/auto` adaptive router, which automatically selects an upstream model per request based on your workspace-level strategy configured in OrcaRouter's routing console.

How do I set up and run this OrcaRouter example?

Get an API key from OrcaRouter, set the `ORCAROUTER_API_KEY` environment variable, then run `npx promptfoo@latest init --example provider-orcarouter`, navigate to the created directory, and execute `npx promptfoo@latest eval`.

When should I use this example instead of other approaches?

Use this example when you want to evaluate either a specific upstream model or OrcaRouter's adaptive routing strategy through a single OpenAI-compatible endpoint and API key, rather than managing separate credentials for each upstream provider.

Does this example show all available models in OrcaRouter?

No, this example demonstrates one named model (`openai/gpt-4o-mini`) plus the auto-router; the full model catalog is available in OrcaRouter's own model listing.

Prompt Chain

Evaluate Prompts Across Multiple LLMs

Name: Provider Orcarouter
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example for OrcaRouter, an OpenAI-compatible adaptive routing gateway to multiple upstream models.

Copy chain

Works with openai

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated 5 days ago

Version 0.121.20

Models

gpt 4o

Add to Favorites

Why it matters

Streamline prompt evaluation by routing requests to multiple LLM providers through a single, unified endpoint. This enables comprehensive testing and comparison of prompt performance across different models.

Outcomes

What it gets done

Route prompts to various OpenAI-compatible models.

Compare and analyze prompt outputs from different LLMs.

Automate prompt testing and validation.

Integrate with OrcaRouter for adaptive LLM selection.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-provider-orcarouter | bash

Steps

Steps in the chain

Get API key from OrcaRouter

Set API key environment variable

Run the evaluation

Overview

Provider Orcarouter

A promptfoo example for OrcaRouter, an OpenAI-compatible adaptive routing gateway that evaluates prompts against a pinned upstream model or an auto-selected one per a workspace routing strategy. Use to evaluate prompts through OrcaRouter's routing layer, pinned or adaptive. Not usable without an OrcaRouter account and API key.

What it does

This promptfoo example calls OrcaRouter, an OpenAI-compatible adaptive routing gateway, to evaluate prompts against multiple upstream models through a single endpoint. It shows two modes: calling a specific upstream model directly (openai/gpt-4o-mini) through OrcaRouter, and using the orcarouter:orcarouter/auto adaptive router, which picks an upstream per request according to a workspace-level strategy configured in OrcaRouter's routing console.

npx promptfoo@latest init --example provider-orcarouter
cd provider-orcarouter

When to use - and when NOT to

Use it to evaluate prompts through OrcaRouter's routing layer - either pinned to one specific upstream model or left to the adaptive router to pick per request. It is not usable without an OrcaRouter account and API key.

Inputs and outputs

Input is an ORCAROUTER_API_KEY environment variable, set after getting an API key from OrcaRouter. Output is the evaluation results from whichever model handled each request - the pinned upstream model, or whatever the adaptive router selected.

Integrations

Calls OrcaRouter's OpenAI-compatible API, which itself proxies requests to multiple upstream models according to a workspace routing strategy.

Who it's for

Teams testing prompts against multiple upstream models through one adaptive-routing endpoint instead of integrating each provider separately.

Source README

provider-orcarouter (OrcaRouter Provider)

This example shows how to use OrcaRouter, an OpenAI-compatible adaptive routing gateway, to evaluate prompts against multiple upstream models through a single endpoint.

You can run this example with:

npx promptfoo@latest init --example provider-orcarouter
cd provider-orcarouter

Setup

Get your API key from OrcaRouter.

Set your API key:

export ORCAROUTER_API_KEY=your_api_key_here

Run the evaluation:
```
npx promptfoo@latest eval
```

What this example does

This example demonstrates:

Calling a specific upstream model (openai/gpt-4o-mini) through OrcaRouter.
Using the orcarouter:orcarouter/auto adaptive router, which picks an upstream per request according to the workspace-level strategy configured in the routing console.

For the full model catalog, see orcarouter.ai/models. For provider docs, see the OrcaRouter provider page.

FAQ

Common questions

Discussion