Integrate Vercel AI SDK with Promptfoo
Reports dynamically-constructed prompts from the Vercel AI SDK to promptfoo, so the real sent prompt is debuggable.
Why it matters
Dynamically construct prompts for your AI applications using the Vercel AI SDK, enhanced by promptfoo's reporting capabilities.
Outcomes
What it gets done
Demonstrate Vercel AI SDK integration
Utilize promptfoo for prompt reporting
Enable dynamic prompt construction
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/pfoo-ai-sdk | bash Steps
Steps in the chain
Overview
Ai Sdk
Demonstrates dynamic prompt construction with the Vercel AI SDK and promptfoo's prompt reporting feature, so the actual persona/task/RAG-context-driven prompt is visible and assertable. Use when debugging or asserting against a dynamically-constructed prompt rather than an unresolved template variable in promptfoo.
What it does
This promptfoo example demonstrates dynamic prompt construction using the Vercel AI SDK combined with promptfoo's provider prompt reporting feature. Modern LLM applications dynamically construct prompts from system instructions tailored to the task, runtime-selected few-shot examples, RAG-retrieved context, and user preferences or safety guardrails - without prompt reporting, promptfoo would only show the unresolved template like {{topic}}, making it impossible to debug what was actually sent or assert against the real prompt content. The provider reports the actual prompt via a prompt field on its return value:
return {
output: result.text,
prompt: [
{ role: 'system', content: dynamicSystemPrompt },
{ role: 'user', content: dynamicUserPrompt },
],
};
Four features are demonstrated: multiple personas (expert, coder, analyst, each with a different system prompt), task types (explain, compare, troubleshoot, each with a different structure), context injection (RAG-style context added to prompts), and template filling (variables like {{domain}} and {{audience}} filled dynamically).
When to use - and when NOT to
Use this example when debugging or asserting against the actual dynamically-constructed prompt sent to a provider - persona-based system prompts, task-specific structures, RAG context injection - rather than only seeing the unresolved template variable in promptfoo's UI.
Inputs and outputs
Run via npx promptfoo@latest init --example integration-vercel/ai-sdk, npm install, export OPENAI_API_KEY=sk-..., then npx promptfoo@latest eval and npx promptfoo@latest view. Clicking any result reveals "Actual Prompt Sent" with the full constructed prompt. A worked example shows input vars (topic: quantum entanglement, persona: expert, domain: quantum physics, audience: college students) producing a full system/user prompt pair with a persona-specific communication style and a structured request covering core concepts, real-world applications, and common misconceptions. Three files structure the example: aiSdkProvider.mjs (the Vercel AI SDK provider with dynamic prompt construction), promptfooconfig.yaml (test cases across personas and task types), and package.json (dependencies ai and @ai-sdk/openai).
Integrations
The prompt-reporting pattern is shown adapted to two other frameworks: LangChain (report prompt.format(input) alongside the chain's output) and a custom RAG pipeline (retrieve context, build fullPrompt, report it alongside the generated output). If Vercel AI SDK's experimental_telemetry is enabled for tool-calling workflows, promptfoo's trajectory assertions can normalize the SDK's tool-call spans from the ai.toolCall.name attribute plus the matching ai.toolCall.args, ai.toolCall.arguments, or ai.toolCall.input attributes.
Who it's for
Engineers building LLM applications with dynamic, persona- or context-dependent prompts who need promptfoo to show and assert on the actual sent prompt rather than an unresolved template variable, across the Vercel AI SDK, LangChain, or a custom RAG pipeline.
Source README
integration-vercel/ai-sdk (Vercel AI SDK Provider)
Demonstrates dynamic prompt construction using the Vercel AI SDK with promptfoo's provider prompt reporting feature.
Why This Matters
Modern LLM applications dynamically construct prompts with:
- System instructions tailored to the task
- Few-shot examples selected at runtime
- Retrieved context from RAG pipelines
- User preferences and safety guardrails
Without prompt reporting, promptfoo shows {{topic}} as the prompt, making it impossible to debug what was actually sent or run assertions on the real prompt content.
How It Works
The provider reports the actual prompt it sent using the prompt field:
return {
output: result.text,
prompt: [
{ role: 'system', content: dynamicSystemPrompt },
{ role: 'user', content: dynamicUserPrompt },
],
};
Features Demonstrated
| Feature | Description |
|---|---|
| Multiple personas | expert, coder, analyst with different system prompts |
| Task types | explain, compare, troubleshoot with different structures |
| Context injection | RAG-style context added to prompts |
| Template filling | Variables like {{domain}}, {{audience}} filled dynamically |
Running the Example
npx promptfoo@latest init --example integration-vercel/ai-sdk
cd integration-vercel/ai-sdk
npm install
export OPENAI_API_KEY=sk-...
npx promptfoo@latest eval
npx promptfoo@latest view
What You'll See
In the promptfoo UI, click any result to see "Actual Prompt Sent" showing the full dynamically-constructed prompt instead of just {{topic}}.
Input:
vars:
topic: quantum entanglement
persona: expert
domain: quantum physics
audience: college students
Actual Prompt Sent:
System: You are a world-class expert in quantum physics.
Your communication style:
- Clear and precise explanations
- Use analogies for complex concepts
- Include concrete examples
- Acknowledge limitations honestly
Your audience: college students
User: Explain quantum entanglement in a way that's accessible and engaging.
Focus on:
1. Core concepts and why they matter
2. Real-world applications
3. Common misconceptions to avoid
Files
| File | Description |
|---|---|
aiSdkProvider.mjs |
Provider using Vercel AI SDK with dynamic prompt construction |
promptfooconfig.yaml |
Test cases showcasing different personas and task types |
package.json |
Dependencies (ai, @ai-sdk/openai) |
Adapting for Your Use Case
The pattern works with any framework:
// LangChain
const chain = prompt.pipe(model);
const result = await chain.invoke(input);
return {
output: result,
prompt: prompt.format(input),
};
// Custom RAG
const context = await retrieveContext(query);
const fullPrompt = `Context: ${context}\n\nQuestion: ${query}`;
const result = await llm.generate(fullPrompt);
return {
output: result,
prompt: fullPrompt,
};
Learn More
Tool Telemetry
If you enable Vercel AI SDK experimental_telemetry for tool-calling workflows, Promptfoo trajectory assertions can normalize the SDK's tool-call spans from ai.toolCall.name plus the matching ai.toolCall.args, ai.toolCall.arguments, or ai.toolCall.input attributes.
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.