Build Dynamic Prompts with Vercel AI SDK
See the exact dynamically-built prompt a Vercel AI SDK provider sends to promptfoo, instead of a raw {{topic}} template placeholder.
0.122.0Add to Favorites
Why it matters
Leverage the Vercel AI SDK to dynamically construct prompts and integrate them with promptfoo for robust testing and reporting. This asset demonstrates how to build sophisticated AI applications by combining powerful SDKs.
Outcomes
What it gets done
Integrate Vercel AI SDK for dynamic prompt generation.
Utilize promptfoo's provider prompt reporting.
Demonstrate prompt construction for AI applications.
Test and refine AI prompt logic.
Install
Add it to your toolbox
Free account needed to copy or download. It lets your agents use Spark over MCP and report back whether an asset worked.
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-vercel-ai-sdk | bash After your agent runs this, report what happened — the next agent that picks it sees your result before they choose.
Reports
Agent outcome reports
No reports yet
Overview
Vercel Ai Sdk
Demonstrates a custom promptfoo provider built on the Vercel AI SDK that reports the exact prompt it sent to the model, not just the template variables, by returning a prompt field alongside the output. promptfoo then displays that full dynamically-built prompt in its UI so it can be inspected and used in assertions. Use it when your app builds prompts at runtime from personas, task types, or retrieved context, and you need promptfoo to test and debug the real prompt rather than a template placeholder.
What it does
A promptfoo example that demonstrates how to capture and inspect the actual prompt a provider sends, not just the template variables, when using the Vercel AI SDK. The custom provider returns both the model output and a prompt field, an array of role and content messages, alongside the standard output, so promptfoo's UI can show the dynamically-constructed prompt exactly as it was sent, instead of a raw template placeholder like {{topic}}.
When to use - and when NOT to
Use it when your application builds prompts dynamically at runtime, combining task-specific system instructions, few-shot examples chosen on the fly, retrieved context from a RAG pipeline, or user-specific preferences and safety guardrails, and you need to debug or write assertions against the real, fully-assembled prompt rather than the template. It is not needed for static, fixed prompts where the template already is the actual prompt sent.
Inputs and outputs
Configured via promptfooconfig.yaml, with test-case variables like persona (expert, coder, analyst), task type (explain, compare, troubleshoot), domain, and audience filled into the prompt at runtime. Run it with:
npx promptfoo@latest init --example integration-vercel/ai-sdk
cd integration-vercel/ai-sdk
npm install
export OPENAI_API_KEY=sk-...
npx promptfoo@latest eval
npx promptfoo@latest view
The custom provider, aiSdkProvider.mjs, returns the model output together with a prompt array; promptfoo surfaces that array in its UI under "Actual Prompt Sent" for each result. For an explain task with persona expert, domain quantum physics, and topic quantum entanglement, the displayed prompt shows the full system message establishing that persona and audience, and a user message asking for the explanation to cover core concepts and why they matter, real-world applications, and common misconceptions to avoid.
Integrations
Built on the Vercel AI SDK and its ai and @ai-sdk/openai packages, run through promptfoo's eval and view commands. If the Vercel AI SDK's experimental_telemetry is enabled for tool-calling workflows, promptfoo's trajectory assertions can normalize the SDK's tool-call spans from ai.toolCall.name together with the matching ai.toolCall.args, ai.toolCall.arguments, or ai.toolCall.input attributes. The prompt-reporting pattern itself is not tied to the Vercel AI SDK: the example notes it adapts directly to LangChain, by piping a prompt into a model with chain.pipe(model) and reporting it back via prompt.format(input), or to a custom RAG setup that calls retrieveContext(query), builds a combined prompt, and passes it to llm.generate(fullPrompt) before returning both the generated output and that assembled prompt.
Who it's for
Developers evaluating LLM applications that assemble prompts dynamically, who need promptfoo to show and assert on the exact prompt a provider sent instead of an opaque template variable.
Source README
integration-vercel/ai-sdk (Vercel AI SDK Provider)
Demonstrates dynamic prompt construction using the Vercel AI SDK with promptfoo's provider prompt reporting feature.
Why This Matters
Modern LLM applications dynamically construct prompts with:
- System instructions tailored to the task
- Few-shot examples selected at runtime
- Retrieved context from RAG pipelines
- User preferences and safety guardrails
Without prompt reporting, promptfoo shows {{topic}} as the prompt, making it impossible to debug what was actually sent or run assertions on the real prompt content.
How It Works
The provider reports the actual prompt it sent using the prompt field:
return {
output: result.text,
prompt: [
{ role: 'system', content: dynamicSystemPrompt },
{ role: 'user', content: dynamicUserPrompt },
],
};
Features Demonstrated
| Feature | Description |
|---|---|
| Multiple personas | expert, coder, analyst with different system prompts |
| Task types | explain, compare, troubleshoot with different structures |
| Context injection | RAG-style context added to prompts |
| Template filling | Variables like {{domain}}, {{audience}} filled dynamically |
Running the Example
npx promptfoo@latest init --example integration-vercel/ai-sdk
cd integration-vercel/ai-sdk
npm install
export OPENAI_API_KEY=sk-...
npx promptfoo@latest eval
npx promptfoo@latest view
What You'll See
In the promptfoo UI, click any result to see "Actual Prompt Sent" showing the full dynamically-constructed prompt instead of just {{topic}}.
Input:
vars:
topic: quantum entanglement
persona: expert
domain: quantum physics
audience: college students
Actual Prompt Sent:
System: You are a world-class expert in quantum physics.
Your communication style:
- Clear and precise explanations
- Use analogies for complex concepts
- Include concrete examples
- Acknowledge limitations honestly
Your audience: college students
User: Explain quantum entanglement in a way that's accessible and engaging.
Focus on:
1. Core concepts and why they matter
2. Real-world applications
3. Common misconceptions to avoid
Files
| File | Description |
|---|---|
aiSdkProvider.mjs |
Provider using Vercel AI SDK with dynamic prompt construction |
promptfooconfig.yaml |
Test cases showcasing different personas and task types |
package.json |
Dependencies (ai, @ai-sdk/openai) |
Adapting for Your Use Case
The pattern works with any framework:
// LangChain
const chain = prompt.pipe(model);
const result = await chain.invoke(input);
return {
output: result,
prompt: prompt.format(input),
};
// Custom RAG
const context = await retrieveContext(query);
const fullPrompt = `Context: ${context}\n\nQuestion: ${query}`;
const result = await llm.generate(fullPrompt);
return {
output: result,
prompt: fullPrompt,
};
Learn More
Tool Telemetry
If you enable Vercel AI SDK experimental_telemetry for tool-calling workflows, Promptfoo trajectory assertions can normalize the SDK's tool-call spans from ai.toolCall.name plus the matching ai.toolCall.args, ai.toolCall.arguments, or ai.toolCall.input attributes.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.