Evaluate LLM Agent Quality

This skill covers testing and benchmarking LLM agents: behavioral testing, capability assessment, reliability metrics, regression testing, and production monitoring - explicitly noting that even top agents achieve less than 50% on real-world benchmarks. It draws on primary tools like AgentBench (multi-environment benchmark, ICLR 2024), tau-bench (Sierra's real-world agent benchmark), ToolEmu (risky tool-use behavior detection), and LangSmith (tracing/evaluation), with Braintrust and PromptFoo as alternatives for production-monitoring or prompt-level focus respectively.

It documents four evaluation patterns - statistical test evaluation (running tests multiple times and analyzing pass-rate distributions for stochastic agents), behavioral contract testing, adversarial testing, and a regression testing pipeline - and four "sharp edges" agents commonly hit: (1) an agent scoring well on benchmarks but failing in production, because benchmarks have known answer patterns while production has messy, long-tail inputs - fixed by testing on real (anonymized) production samples, adversarial variants, and load; (2) the same test passing sometimes and failing other times because LLM outputs are stochastic and tests expect deterministic behavior - fixed by running each test multiple times and computing a flakiness score before trusting a pass/fail; (3) an agent optimized for a single metric rather than the actual task, detected via multi-dimensional evaluation (correctness, helpfulness, safety, efficiency, user preference) that flags high variance across dimensions as a gaming signal; and (4) test data accidentally leaking into training data, system prompts, or RAG retrieval, artificially inflating scores - detected via similarity checks against training data, system prompt scanning, memorization probes, and RAG-retrieval leakage checks.

Use this when testing or benchmarking an LLM agent's capability, reliability, or production-readiness, or when investigating flaky CI, suspiciously perfect scores, or a gap between benchmark performance and real-world user satisfaction.

This skill explicitly does NOT cover model training evaluation (loss, perplexity), fairness/bias testing, or user experience testing - its focus is agent capability and reliability, functional and behavioral testing specifically. It also assumes familiarity with testing methodologies, statistical analysis basics, and LLM behavior patterns as prerequisites.

Inputs: an agent under test, a benchmark or test suite (existing or custom), and (for leakage detection) the agent's training data and system prompt.

Outputs: pass rates with confidence intervals, flakiness scores per test, multi-dimensional evaluation reports with gaming detection, production-readiness gap reports, and data-leakage reports flagging training-data, system-prompt, memorization, or RAG-retrieval leaks.

AgentBench, tau-bench, ToolEmu, LangSmith (primary tools); Braintrust and PromptFoo as alternatives depending on whether production monitoring or prompt-level testing is the priority.

Teams building or shipping LLM agents who need rigorous capability/reliability testing beyond a simple benchmark score - particularly to catch benchmark-production gaps, test flakiness, metric gaming, and data leakage before they undermine confidence in the agent.