Evaluate Chatbot Conversation Relevance

This prompt assertion is designed to evaluate whether chatbot responses remain relevant throughout a conversation. It helps ensure that a language model can maintain context and stay on topic across multiple turns of dialogue.

Use this assertion when you need to rigorously test a chatbot's ability to maintain conversational coherence and relevance over extended interactions. It is particularly useful for applications where consistent topic adherence is critical, such as customer support bots, virtual assistants, or any conversational AI that engages in multi-turn dialogues.

Do NOT use this assertion if you are only evaluating single-turn responses or if the primary concern is factual accuracy independent of conversational flow. It is also not suitable for testing creative writing generation where topic drift might be intentional or acceptable.

The primary input is a multi-turn conversation, typically structured as a series of user prompts and model responses. The assertion analyzes these turns to determine if each model response is relevant to the ongoing conversation. The output is a pass/fail result for the conversation-relevance assertion, indicating whether the chatbot maintained relevance throughout the provided dialogue.

This assertion is part of the promptfoo framework. promptfoo is a tool for testing and evaluating LLM applications. Further integrations are not specified in the source material.

This assertion is for developers, QA engineers, and AI researchers working with conversational AI systems. It is particularly beneficial for those building or refining chatbots and virtual assistants where maintaining a coherent and relevant dialogue is a key performance indicator. Alternatives might involve manual review, which is less scalable, or simpler relevance checks that don't account for multi-turn context.