Compare Open Source LLMs on Key Tasks
Promptfoo example comparing DeepSeek, Mistral, Llama, and Qwen on factual tasks through a single OpenRouter key.
Maintainer of this project? Claim this page to edit the listing.
code-scan-action-0.1Add to Favorites
Why it matters
Evaluate and compare the performance of leading open-source large language models (LLMs) like DeepSeek, Mistral, Llama, and Qwen. This asset helps you understand their capabilities across various factual tasks.
Outcomes
What it gets done
Benchmark LLM performance on factual assertions.
Compare outputs from multiple open-source models.
Utilize OpenRouter for model access and comparison.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-compare-open-source-models | bash Steps
Steps in the chain
Overview
Compare Open Source Models
This example compares DeepSeek, Mistral, Llama, and Qwen on factual accuracy through a single OpenRouter API key. Use it when comparing DeepSeek, Mistral, Llama, and Qwen on factual accuracy without configuring separate provider API keys.
What it does
This example compares DeepSeek, Mistral, Llama, and Qwen on various tasks using factual assertions, accessed through a single OpenRouter API key.
When to use - and when NOT to
Use it when you want a working comparison of four open-source model families on factual accuracy, without configuring separate API keys for each provider. Do not use it if you need models outside this set - the example is specifically scoped to DeepSeek, Mistral, Llama, and Qwen.
Inputs and outputs
Requires OPENROUTER_API_KEY. Running the evaluation scores each model's responses against factual assertions defined in the config, viewable via promptfoo view.
Integrations
Set up via npx promptfoo@latest init --example compare-open-source-models, then npx promptfoo@latest eval and npx promptfoo@latest view. A companion guide covers the comparison methodology in more depth.
Who it's for
Teams evaluating open-source model families - DeepSeek, Mistral, Llama, and Qwen - on factual accuracy through a single OpenRouter key rather than juggling separate provider accounts.
Source README
compare-open-source-models (Open-Source Model Comparison)
This example compares DeepSeek, Mistral, Llama, and Qwen on various tasks with factual assertions using OpenRouter.
See the guide for more details.
Setup
npx promptfoo@latest init --example compare-open-source-models
cd compare-open-source-models
Set the required environment variable:
export OPENROUTER_API_KEY=your-openrouter-key
Run
npx promptfoo@latest eval
View
npx promptfoo@latest view
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.