Compare LLM Performance
Promptfoo starting point for comparing Llama (via Replicate) and GPT models side by side on your own test cases.
Why it matters
Evaluate and compare the performance of different large language models (LLMs) like Llama and GPT on various tasks. Understand their strengths and weaknesses to choose the best model for your needs.
Outcomes
What it gets done
Benchmark LLM responses
Analyze model outputs
Identify performance differences
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-compare-llama-vs-gpt | bash Steps
Steps in the chain
Overview
Compare Llama Vs Gpt
This is a minimal promptfoo starting point for comparing Meta Llama, via Replicate, and GPT model responses side by side on custom test cases. Use it as a bare-bones starting point for comparing Llama and GPT models; not a ready-made comparison task.
What it does
This example runs a promptfoo evaluation comparing Meta Llama, accessed via Replicate, and OpenAI GPT models side by side on the same test cases.
When to use - and when NOT to
Use it as a starting point for comparing Llama and GPT model responses on your own prompts and test cases. Do not use it expecting a pre-built comparison task - the source provides only the generic setup and run steps, with prompts meant to be edited in prompts.txt for your own comparison.
Inputs and outputs
Requires OPENAI_API_KEY and REPLICATE_API_TOKEN. Inputs are the prompts in prompts.txt and the config in promptfooconfig.yaml; running promptfoo eval produces side-by-side results, viewable with promptfoo view.
Integrations
Set up via npx promptfoo@latest init --example compare-llama-vs-gpt, then edit prompts.txt and promptfooconfig.yaml before running the evaluation.
Who it's for
Developers who want a bare-bones promptfoo starting point for comparing Llama and GPT model responses on their own test cases.
Source README
compare-llama-vs-gpt (Llama Gpt Comparison)
You can run this example with:
npx promptfoo@latest init --example compare-llama-vs-gpt
cd compare-llama-vs-gpt
Usage
To get started, set your OPENAI_API_KEY and REPLICATE_API_TOKEN environment variables.
Next, change a few of the prompts in prompts.txt and edit promptfooconfig.yaml.
Then run:
promptfoo eval
Afterwards, you can view the results by running promptfoo view
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.