This promptfoo example demonstrates dynamic prompt construction using a custom provider (file://./aiSdkProvider.mjs) built on the Vercel AI SDK. Instead of a fixed prompt template, the provider builds the actual prompt sent to the LLM (gpt-4o-mini) based on variables like persona (expert, coder, analyst), task_type (explain, compare, troubleshoot), domain, audience, and optional RAG-style context, and reports back the constructed prompt.

Use this example when your application composes prompts dynamically from structured inputs (persona, task type, audience, retrieved context) rather than filling a single static template, and you want to evaluate that composition logic alongside the resulting LLM output. It requires a custom provider script to implement the prompt-construction logic; it is not a fixed-prompt eval and needs adaptation if your composition rules differ.

The shared prompt is just '{{topic}}', with all the real construction logic implemented inside aiSdkProvider.mjs based on additional test variables:

providers:
  - id: file://./aiSdkProvider.mjs
    config:
      model: gpt-4o-mini
      temperature: 0.7

tests:
  - description: Explain with context (RAG simulation)
    vars:
      topic: transformer architecture
      persona: expert
      task_type: explain
      domain: machine learning
      audience: ML engineers
      context: |
        From "Attention Is All You Need" (2017):
        The Transformer uses self-attention to compute representations...
    assert:
      - type: icontains
        value: attention
      - type: llm-rubric
        value: references or builds upon the provided context

Other tests vary persona/task_type combinations - a coder persona explaining async/await, an analyst comparing SQL vs NoSQL with a structured pros/cons format, and an expert troubleshooting Node.js memory leaks - each checked with a mix of icontains keyword checks and llm-rubric quality checks.

Uses the Vercel AI SDK inside a custom promptfoo provider script (aiSdkProvider.mjs) to call gpt-4o-mini, with promptfoo's icontains and llm-rubric assertion types for grading.

Developers building applications with dynamic, persona/task-aware prompt construction (e.g. multi-mode assistants or RAG pipelines) who want to evaluate both the constructed prompt logic and the resulting model output using the Vercel AI SDK.