Evaluate Conversation Relevance

This promptfoo example demonstrates the conversation-relevance assertion, which evaluates whether each response in a multi-turn conversation stays relevant to the context and prior messages, using a sliding-window approach across conversation segments. It ships four test cases: a single-turn evaluation (a basic travel-to-Paris query-response pair), a multi-turn travel-planning conversation where every response should be relevant, a wedding-planning conversation with a deliberately off-topic stock-market comment inserted to show detection of an irrelevant response, and a technical-support conversation graded at a high 0.95 relevance threshold.

Scoring works window by window: for a 5-message conversation with window size 3, Window 1 covers message 1 only, Window 2 covers messages 1-2, Window 3 covers messages 1-3, Window 4 covers messages 2-4, and Window 5 covers messages 3-5 - each window judging whether the last assistant response in it is relevant given the context available in that window. The final score is the number of relevant windows divided by the total number of windows. Results report three things: the score itself (the proportion of windows deemed relevant), a pass/fail verdict against the configured threshold, and a reason explaining why any given response was found irrelevant.

npx promptfoo@latest init --example eval-conversation-relevance
cd eval-conversation-relevance

Use it to catch a multi-turn chatbot drifting off-topic - general assistants, travel or wedding planning bots, or technical support where relevance matters. The example's own tips: use lower thresholds (0.7-0.8) for general conversations, higher thresholds (0.9-0.95) for specialized domains, adjust window size to conversation complexity, and use a more capable grading model (GPT-4) for complex conversations. It is specifically about relevance drift across turns, so it adds nothing for single-shot Q&A that has no conversation history to drift within.

Input is an OPENAI_API_KEY, promptfoo installed globally (npm install -g promptfoo), and an eval config with the conversation-relevance assertion set to a threshold (0-1, default 0.5) and config.windowSize (messages per window, default 5) - run with promptfoo eval. Output is the score/pass-fail/reason breakdown described above, generated per conversation window rather than once for the whole conversation.

Relevance judgments are made by a grading model - OpenAI by default, overridable per the assertion's provider option to use a different or more capable model for complex conversations.

Teams building multi-turn chatbots - travel planning, wedding planning, technical support, or general assistants - who need automated detection of topic drift partway through a conversation.