Evaluate Conversation Relevance

This example demonstrates promptfoo's conversation-relevance assertion, which evaluates whether each response in a conversation stays relevant to the context and prior messages, using a sliding-window approach. It includes four test cases: a single-turn evaluation of relevance for a query-response pair about travel to Paris, a multi-turn travel-planning conversation where every response should be relevant, a wedding-planning conversation containing one deliberately off-topic response about the stock market to demonstrate detection of an irrelevant reply in the middle of an otherwise-relevant conversation, and a high-quality technical support conversation scored with a 0.95 threshold. Configuration options include threshold (minimum passing score, 0-1), config.windowSize (messages per sliding window, default 5), and an overridable grading provider. Scoring works by evaluating the last assistant response in each window for relevance; for a 5-message conversation with window size 3, Window 1 covers message 1 only, Window 2 covers messages 1-2, Window 3 covers messages 1-3, Window 4 slides to messages 2-4, and Window 5 slides to messages 3-5 - each window scoring whether the last assistant response within it is relevant. The final score is the number of relevant windows divided by the total number of windows.

Use this when evaluating a chatbot or conversational agent's tendency to stay on-topic across multiple turns. The example's own tips recommend lower thresholds (0.7-0.8) for general conversations and higher thresholds (0.9-0.95) for specialized domains like technical support, adjusting window size based on conversation complexity, and using a more capable grading model (e.g. GPT-4) for judging complex conversations.

Not needed for single-turn, stateless prompts with no conversational context to drift from.

Requires OPENAI_API_KEY (or npm install -g promptfoo). Input is promptfooconfig.yaml with conversation-relevance assertions across the four test cases. Output is a normal promptfoo eval run, reporting a relevance score (the proportion of conversation windows deemed relevant), pass/fail against the threshold, and a reason when responses are found irrelevant.

Uses an overridable grading provider (defaulting to OpenAI via OPENAI_API_KEY) to judge relevance per sliding window; more capable models are recommended for grading complex conversations.

Teams evaluating chatbots or multi-turn conversational agents that need to detect topic drift, with different relevance bars for general versus specialized domains. The provider config option lets teams override the default grading model entirely, which pairs with the tip to use a more capable model for judging complex, multi-topic conversations.

npx promptfoo@latest init --example eval-conversation-relevance
cd eval-conversation-relevance