Compare LLM Performance
Promptfoo starting point for comparing local Ollama-hosted Phi3 and Llama3 models side by side on your own test cases.
Why it matters
Evaluate and compare the performance of different large language models (LLMs) like Phi and Llama. This asset helps you understand which model is better suited for specific tasks by running comparative tests.
Outcomes
What it gets done
Set up comparative tests for LLMs.
Analyze and summarize LLM outputs.
Identify strengths and weaknesses of different models.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-compare-phi-vs-llama | bash Steps
Steps in the chain
Overview
Compare Phi Vs Llama
This is a minimal promptfoo starting point for comparing local, Ollama-hosted Phi3 and Llama3 model responses side by side, with no API keys required. Use it as a bare-bones, local starting point for comparing Phi3 and Llama3; not a ready-made comparison task.
What it does
This example runs a promptfoo evaluation comparing Microsoft's Phi3 and Meta's Llama3 models, both run locally via Ollama, side by side on the same test cases.
When to use - and when NOT to
Use it as a starting point for comparing Phi3 and Llama3 responses locally, with no API keys or cloud costs required. Do not use it expecting a pre-built comparison task - the source provides only the generic setup and run steps, and you edit the config to define your own comparison.
Inputs and outputs
Requires Ollama installed with both models pulled locally (ollama pull llama3, ollama pull phi3). Inputs are whatever you define in promptfooconfig.yaml; running promptfoo eval produces side-by-side results, viewable with promptfoo view.
Integrations
Set up via npx promptfoo@latest init --example compare-phi-vs-llama, pull both models locally, then edit promptfooconfig.yaml before running the evaluation.
Who it's for
Developers who want a bare-bones, local, no-API-key promptfoo starting point for comparing Phi3 and Llama3 model responses on their own test cases.
Source README
compare-phi-vs-llama (Phi Vs Llama)
You can run this example with:
npx promptfoo@latest init --example compare-phi-vs-llama
cd compare-phi-vs-llama
Usage
To get started:
ollama pull llama3
ollama pull phi3
Next, edit promptfooconfig.yaml.
Then run:
promptfoo eval
Afterwards, you can view the results by running promptfoo view
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.