Evaluate RAG Systems with Promptfoo

This promptfoo example evaluates a corporate internal chatbot (openai:gpt-4.1-mini) that answers HR-policy questions using retrieved document context, scoring each response across five distinct RAG quality dimensions: factuality (does the answer match a known fact), answer-relevance (does it address the query), context-recall (did the answer use the relevant facts present in the context), context-relevance (was the retrieved context itself relevant), and context-faithfulness (does the answer stick to what the context actually says).

Use this example when you need to evaluate a retrieval-augmented generation pipeline holistically - not just whether the final answer is correct, but whether the retrieved context was relevant and whether the answer stayed faithful to it. It is a strong template when you have separate context and query variables per test, as a typical RAG setup does; it is not applicable to prompts without a distinct retrieved-context input.

The shared prompt injects both query and context (loaded from files like file://docs/reimbursement.md) into a corporate-chatbot persona. Each test combines a contains or factuality check on the answer with four threshold-based RAG metrics:

  - vars:
      query: What is the max purchase that doesn't require approval?
      context: file://docs/reimbursement.md
    assert:
      - type: contains
        value: '$500'
      - type: factuality
        value: the employee's manager is responsible for approvals
      - type: answer-relevance
        threshold: 0.9
      - type: context-recall
        threshold: 0.9
        value: max purchase price without approval is $500. Talk to Fred before submitting anything.
      - type: context-relevance
        threshold: 0.9
      - type: context-faithfulness
        threshold: 0.9

A second test on maternity leave uses a deliberately absurd context-recall reference value ("4 months of maternity leave, unless you are an elephant, in which case you get 22 months") to illustrate how the recall metric compares the answer against a specific expected-facts string.

Runs against openai:gpt-4.1-mini with file-loaded context documents, and uses promptfoo's contains, factuality, answer-relevance, context-recall, context-relevance, and context-faithfulness assertion types, several with a 0.9 pass threshold.

Teams building RAG-based internal tools (HR bots, support assistants, knowledge-base chatbots) who want a template covering the full set of promptfoo's RAG-specific quality metrics rather than just final-answer correctness.