Spark
Sign In
Prompt Chain

Evaluate Prompts Across Multiple LLMs

Evaluate prompts against multiple LLM providers via a single endpoint using OrcaRouter for adaptive routing and comprehensive testing.

Copy chain
Works with openai
Promptfoo
Maintainer?
54
Spark score
out of 100
Updated 2 days ago
Version code-scan-action-0.1
Models
gpt 4o
Add to Favorites

Why it matters

Streamline prompt evaluation by routing requests to multiple LLM providers through a single, unified endpoint. This enables comprehensive testing and comparison of prompt performance across different models.

Outcomes

What it gets done

01

Route prompts to various OpenAI-compatible models.

02

Compare and analyze prompt outputs from different LLMs.

03

Automate prompt testing and validation.

04

Integrate with OrcaRouter for adaptive LLM selection.

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-provider-orcarouter | bash

Capabilities

What this chain does

Generate code

Writes source code or scripts from a description.

Review code

Analyzes code for bugs, style issues, and improvements.

Write tests

Creates unit, integration, or end-to-end test cases.

Overview

Provider Orcarouter

What it does

provider-orcarouter (OrcaRouter Provider)

This example shows how to use OrcaRouter, an OpenAI-compatible adaptive routing gateway, to evaluate prompts against multiple upstream models through a single endpoint.

You can run this example with:

npx promptfoo@latest init --example provider-orcarouter
cd provider-orcarouter

Setup

  1. Get your API key from OrcaRouter.

  2. Set your API key:

    export ORCAROUTER_API_KEY=your_api_key_here

  3. Run the evaluation:

    npx promptfoo@latest eval

What this example does

This example demonstrates:

  • Calling a specific upstream model (openai/gpt-4o-mini) through OrcaRouter.
  • Using the orcarouter:orcarouter/auto adaptive router, which picks an upstream per request according to the workspace-level strategy configured in the routing console.

For the full model catalog, see orcarouter.ai/models. For provider docs, see the OrcaRouter provider page.

Source README

provider-orcarouter (OrcaRouter Provider)

This example shows how to use OrcaRouter, an OpenAI-compatible adaptive routing gateway, to evaluate prompts against multiple upstream models through a single endpoint.

You can run this example with:

npx promptfoo@latest init --example provider-orcarouter
cd provider-orcarouter

Setup

  1. Get your API key from OrcaRouter.

  2. Set your API key:

    export ORCAROUTER_API_KEY=your_api_key_here

  3. Run the evaluation:

    npx promptfoo@latest eval

What this example does

This example demonstrates:

  • Calling a specific upstream model (openai/gpt-4o-mini) through OrcaRouter.
  • Using the orcarouter:orcarouter/auto adaptive router, which picks an upstream per request according to the workspace-level strategy configured in the routing console.

For the full model catalog, see orcarouter.ai/models. For provider docs, see the OrcaRouter provider page.

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.