Spark
Sign In
Prompt Chain

Evaluate LLM outputs using GitHub Models API

Test AI model responses using GitHub Models API as provider and grader across OpenAI, Meta, DeepSeek, and Mistral models.

Copy chain
Works with githubopenaimetadeepseekmistral
Promptfoo
Own this? Claim it
82
Spark score
out of 100
Updated 19 days ago
Version 0.121.18
Models
gpt 4odeepseek v3mistral largellama 3
Add to Favorites

Why it matters

Test and evaluate AI model responses by integrating GitHub Models API as both a primary provider and grading mechanism, enabling developers to assess model performance across multiple providers including OpenAI, Meta, DeepSeek, and Mistral through a unified interface.

Outcomes

What it gets done

01

Connect to GitHub Models API using personal access tokens for authentication

02

Run evaluations against multiple models including GPT-4o, GPT-5, Llama, and DeepSeek

03

Grade model outputs using LLM-rubric assertions powered by GitHub Models

04

View and compare evaluation results across different model providers

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-provider-github-models | bash

Steps

Steps in the chain

01
Setup GitHub Token
02
Run the Evaluation
03
View Results

Overview

Provider Github Models

This prompt chain integrates GitHub Models API into Promptfoo's evaluation framework, enabling you to test AI model responses using models hosted on GitHub's marketplace. It supports direct provider usage for generating responses and uses GitHub Models as a grader for assertion validation through llm-rubric configurations. Use this workflow when you need to evaluate OpenAI models (GPT-4o, GPT-5 series, o3-mini, o4-mini) or alternative providers (Meta Llama, DeepSeek-V3, Mistral) through GitHub's unified API without managing separate provider credentials. It works with GitHub Models for both generating test responses and grading them using llm-rubric assertions via the defaultTest.options.provider.text configuration. Do NOT use this if you require models outside GitHub's marketplace or need direct provider APIs with extended configuration options not exposed through GitHub's abstraction layer.

What it does

This prompt chain integrates GitHub Models API into Promptfoo's evaluation framework, enabling you to test AI model responses using models hosted on GitHub's marketplace. It supports direct provider usage for generating responses and uses GitHub Models as a grader for assertion validation through llm-rubric configurations.

When to use - and when NOT to

Use this workflow when you need to evaluate OpenAI models (GPT-4o, GPT-5 series, o3-mini, o4-mini) or alternative providers (Meta Llama, DeepSeek-V3, Mistral) through GitHub's unified API without managing separate provider credentials. It works with GitHub Models for both generating test responses and grading them using llm-rubric assertions via the defaultTest.options.provider.text configuration. Do NOT use this if you require models outside GitHub's marketplace or need direct provider APIs with extended configuration options not exposed through GitHub's abstraction layer.

Inputs and outputs

You provide a GitHub Personal Access Token and a Promptfoo configuration specifying which GitHub Models to test. Set your token as shown:

export GITHUB_TOKEN=ghp_your_token_here

You can then run evaluations and view results using the Promptfoo commands.

Integrations

This workflow integrates with:

  • GitHub Models API: Access to OpenAI models (github:openai/gpt-4o, github:openai/gpt-4o-mini, github:openai/gpt-5, github:openai/gpt-5-mini, github:openai/gpt-5-nano, github:openai/o4-mini, github:openai/o3-mini), Meta models (github:meta/llama-4-scout-17b-16e-instruct), DeepSeek models (github:deepseek/DeepSeek-V3-0324), and Mistral models (github:mistral-ai/mistral-large)
  • Promptfoo evaluation framework: Uses native llm-rubric assertions with GitHub Models as the grading provider
  • GitHub Personal Access Tokens: Authentication via tokens created at https://github.com/settings/tokens

Who it's for

AI engineers and prompt developers who want to benchmark multiple model families through a single API endpoint. Teams already using GitHub for development will find this particularly convenient since it consolidates model access under existing GitHub authentication. It's valuable for organizations evaluating which model provider (OpenAI, Meta, DeepSeek, Mistral) best fits their use case without managing separate API keys for each vendor.

To get started:

npx promptfoo@latest init --example provider-github-models
cd provider-github-models

Then run evaluations with:

promptfoo eval

View results in the web interface:

promptfoo view
Source README

provider-github-models (GitHub Models Provider)

You can run this example with:

npx promptfoo@latest init --example provider-github-models
cd provider-github-models

Setup

Set your GITHUB_TOKEN environment variable. You can create a Personal Access Token at https://github.com/settings/tokens.

export GITHUB_TOKEN=ghp_your_token_here

What This Tests

This example tests the GitHub Models API integration:

  1. Direct Provider Usage: Uses GitHub Models as the main provider (gpt-4o-mini and gpt-5-mini)
  2. GitHub as Grader: Uses GitHub Models for llm-rubric assertions via the defaultTest.options.provider.text configuration

Run the Evaluation

promptfoo eval

View Results

promptfoo view

Available Models

GitHub Models supports various models including:

OpenAI Models:

  • github:openai/gpt-4o, github:openai/gpt-4o-mini
  • github:openai/gpt-5, github:openai/gpt-5-mini, github:openai/gpt-5-nano
  • github:openai/o4-mini, github:openai/o3-mini

Other Providers:

  • github:meta/llama-4-scout-17b-16e-instruct
  • github:deepseek/DeepSeek-V3-0324
  • github:mistral-ai/mistral-large

See the GitHub Models marketplace for the full list.

FAQ

Common questions

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.