Generate and Test Code with Fireworks AI
Promptfoo example comparing three Fireworks AI serverless chat models on summarization, graded via similarity.
Why it matters
This asset helps developers generate, test, and refine code using the Fireworks AI platform. It automates parts of the code development lifecycle, ensuring higher quality output.
Outcomes
What it gets done
Generate code snippets based on prompts.
Execute tests against generated code.
Debug and iterate on code based on test results.
Integrate with Fireworks AI for code generation.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-provider-fireworks | bash Steps
Steps in the chain
Overview
Provider Fireworks
A promptfoo example comparing three Fireworks AI serverless reasoning models on summarization, graded via similarity scoring from a Fireworks embedding model. Use when comparing reasoning-capable serverless models on Fireworks for summarization, with similarity-based grading.
What it does
This example compares three Fireworks AI serverless chat models - gpt-oss-120b, deepseek-v4-pro, and kimi-k2p6 - on a summarization task. Since these are reasoning models, max_tokens is set high enough to leave room for hidden reasoning tokens plus the visible answer. Output is graded with a similar assertion, itself scored by a Fireworks embedding model via the fireworks:embedding: prefix.
When to use - and when NOT to
Use this example when comparing reasoning-capable serverless models on Fireworks for a summarization-style task, including similarity-based grading via a Fireworks embedding model. Serverless models rotate in and out of Fireworks's catalogue over time - if a named model returns a 404, pick a current one from Fireworks's serverless model catalogue instead.
Inputs and outputs
Get a FIREWORKS_API_KEY from fireworks.ai (Settings -> API Keys), set it as an environment variable, then scaffold and run:
npx promptfoo@latest init --example provider-fireworks
cd provider-fireworks
followed by promptfoo eval and promptfoo view.
Integrations
Integrates Fireworks AI's serverless chat models (gpt-oss-120b, deepseek-v4-pro, kimi-k2p6) alongside a Fireworks embedding model referenced via the fireworks:embedding: provider prefix, used to grade output similarity.
Who it's for
Teams comparing reasoning-capable serverless models on Fireworks AI for summarization or similar tasks, who want similarity-based grading via a Fireworks-hosted embedding model rather than a separate embedding provider.
Source README
provider-fireworks (Fireworks AI)
You can run this example with:
npx promptfoo@latest init --example provider-fireworks
cd provider-fireworks
Usage
Set your FIREWORKS_API_KEY environment variable. You can get a key from fireworks.ai, sign in, open Settings -> API Keys, and create one.
Then run:
promptfoo eval
View the results with promptfoo view.
What this shows
- Three Fireworks serverless chat models (
gpt-oss-120b,deepseek-v4-pro,kimi-k2p6) compared on a summarisation task. These are reasoning models, somax_tokensis set high enough to leave room for hidden reasoning tokens plus the visible answer. - A
similarassertion graded by a Fireworks embedding model via thefireworks:embedding:prefix.
Models rotate in and out of the serverless tier - if a model 404s, pick a current one from the serverless catalogue.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.