Evaluate Cerebras LLM Inference API Models
A promptfoo example evaluating Cerebras-hosted Llama models: basic completions, structured JSON output, and tool calling.
Why it matters
Integrate with the Cerebras Inference API to evaluate the performance of Llama and other LLM models. This asset helps developers ensure their models are running efficiently and accurately on Cerebras hardware.
Outcomes
What it gets done
Set up promptfoo for Cerebras provider integration.
Define evaluation prompts and expected outputs.
Run inference tests against Cerebras API endpoints.
Analyze and debug model performance results.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-provider-cerebras | bash Steps
Steps in the chain
Overview
Provider Cerebras
This promptfoo example evaluates Cerebras-hosted Llama models across three configs: basic completions, JSON-schema-enforced structured output for recipe data, and calculator tool-calling with step-by-step reasoning. Use it as a template for evaluating Cerebras-hosted models through promptfoo. It requires a Cerebras API key and is specific to the Cerebras Inference API.
What it does
This is a promptfoo example demonstrating the Cerebras provider, which evaluates models on Cerebras's high-performance Inference API for Llama and other LLMs. It ships three example configurations: basic model evaluation (comparing two Cerebras models on explaining complex concepts simply, measuring clarity, accuracy, and response time), structured outputs (Cerebras's JSON schema enforcement producing consistent recipe data with typed, required fields like cuisine, difficulty, ingredients, and instructions), and tool use (function calling with a calculator tool that solves math problems and explains its steps, e.g. computing 15 x 7 = 105). The basic evaluation's expected output is a side-by-side comparison of how each model explains concepts drawn from different domains, scored on clarity, accuracy, and response time rather than a single pass/fail result.
When to use - and when NOT to
Use it as a starting template for evaluating Cerebras-hosted models - basic completions, JSON-schema-constrained structured output, or function/tool calling - through promptfoo. It requires a Cerebras API key from console.cerebras.ai; it's specific to the Cerebras Inference API rather than a general multi-provider comparison.
Inputs and outputs
Input is a CEREBRAS_API_KEY environment variable (or .env entry) plus the chosen config file. Output depends on which example you run:
promptfoo eval -c promptfooconfig-structured.yaml
for the structured-output example, producing schema-validated recipe JSON; promptfoo eval alone for the basic model comparison; or promptfoo eval -c promptfooconfig-tools.yaml for the tool-use example, where the model calls a calculator tool and explains its reasoning step by step rather than just returning a final number.
Integrations
Built on promptfoo and the Cerebras Inference API, covering models llama-4-scout-17b-16e-instruct (Llama 4 Scout 17B, 16-expert MoE, used in the examples), llama3.1-8b, llama-3.3-70b, and deepSeek-r1-distill-llama-70B in private preview. Cerebras usage is billed on input and output tokens per its own pricing page, positioned as competitive against other high-performance inference services in the market.
The structured-output recipe schema includes fields like cuisine type, difficulty level, prep and cook time, an ingredients list with amounts, and step-by-step cooking instructions - all validated against the defined JSON schema rather than loosely parsed from free text.
Who it's for
Developers evaluating Cerebras-hosted Llama models for inference speed, structured-output reliability, or tool-calling accuracy through promptfoo, without hand-writing the provider configuration themselves. It also links out to Cerebras's own provider documentation, API reference, structured-outputs guide, and tool-use guide for anyone extending past the three bundled examples into their own use case.
Source README
provider-cerebras (Cerebras Example (High-Performance LLM Inference))
This example demonstrates how to use the Cerebras provider with promptfoo to evaluate Cerebras Inference API models, which offer high-performance inference for Llama and other LLM models.
You can run this example with:
npx promptfoo@latest init --example provider-cerebras
cd provider-cerebras
Prerequisites
API Key Setup
- Sign up for an account at Cerebras AI
- Navigate to your account settings to generate an API key
- Set your Cerebras API key as an environment variable:
export CEREBRAS_API_KEY="your-api-key-here"
Alternatively, you can add it to your .env file:
CEREBRAS_API_KEY=your-api-key-here
Example Configurations
This repository contains three example configurations demonstrating different Cerebras features:
1. Basic Model Evaluation (promptfooconfig.yaml)
This configuration evaluates two Cerebras models on their ability to explain complex concepts in simple terms.
promptfoo eval
Expected output: You'll see a comparison of how each model explains concepts from different domains, with metrics on clarity, accuracy, and response time.
2. Structured Outputs (promptfooconfig-structured.yaml)
The structured output example demonstrates Cerebras's JSON schema enforcement capabilities, ensuring the model returns consistent, structured recipe data with proper types and required fields.
promptfoo eval -c promptfooconfig-structured.yaml
Expected output: You'll receive structured JSON outputs for different recipes, with consistent fields like cuisine type, difficulty level, ingredients, and cooking instructions - all following the defined schema.
Example output:
{
"name": "Traditional Pasta Carbonara",
"cuisine": "Italian",
"difficulty": "medium",
"prepTime": 15,
"cookTime": 20,
"ingredients": [
{ "name": "spaghetti", "amount": "400g" },
{ "name": "pancetta", "amount": "150g" },
{ "name": "eggs", "amount": "3 large" },
{ "name": "parmesan cheese", "amount": "50g" }
],
"instructions": [
"Bring a large pot of salted water to boil",
"Cook spaghetti according to package instructions",
"In a separate pan, cook pancetta until crispy",
"In a bowl, whisk eggs and grated parmesan cheese",
"Drain pasta, reserving some pasta water",
"Toss hot pasta with pancetta, then quickly mix in egg mixture",
"Add pasta water as needed to create a silky sauce"
]
}
3. Tool Use (promptfooconfig-tools.yaml)
The tool use example demonstrates Cerebras's function calling capabilities with a calculator tool that the model can use to solve math problems.
promptfoo eval -c promptfooconfig-tools.yaml
Expected output: The model will use the calculator tool to solve math problems and provide step-by-step explanations of the solution process. For example, when given "15 × 7", it will calculate 105 and explain multiplication concepts.
Model Capabilities
Cerebras supports several powerful models:
llama-4-scout-17b-16e-instruct- Llama 4 Scout 17B model with 16 expert MoE (featured in examples)llama3.1-8b- Llama 3.1 8B modelllama-3.3-70b- Llama 3.3 70B modeldeepSeek-r1-distill-llama-70B(private preview)
Pricing & Usage
Cerebras Inference API offers competitive pricing compared to other inference services. Check the official pricing page for the most current rates. Usage is billed based on input and output tokens.
Learn More
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.