Evaluate Agent System Performance

This skill builds evaluation frameworks for agent systems, addressing why agents need different evaluation approaches than traditional software or standard LLM applications - they make dynamic decisions, are non-deterministic between runs, and often lack a single correct answer, so evaluation must judge whether they achieve the right outcome via a reasonable process rather than checking for specific steps.

It reports a key research finding from the BrowseComp evaluation (testing browsing agents' ability to locate hard-to-find information): three factors explain 95% of performance variance - token usage (80%), number of tool calls (10%), and model choice (5%). This implies evaluation should use realistic token budgets rather than unlimited resources, that model upgrades (e.g. to Claude Sonnet 4.5 or GPT-5.2) beat simply doubling token budgets, and that the finding validates multi-agent architectures distributing work across separate context windows.

Its evaluation challenges cover non-determinism and multiple valid paths (different agents may use different tools or explore different numbers of sources to reach the same correct answer - solved via outcome-focused rather than step-checking evaluation), context-dependent failures (agents may succeed on simple queries but fail on complex ones, or fail only after extended interaction as context accumulates - requiring evaluation across complexity levels and extended interactions), and composite quality dimensions (accuracy, completeness, coherence, tool efficiency, and process quality don't move together, so rubrics need multiple weighted dimensions).

Its rubric design covers five example dimensions (factual accuracy, completeness, citation accuracy, source quality, tool efficiency, each rated on an excellent-to-failed scale) converted to 0.0-1.0 numeric scores with appropriate weighting into a weighted overall score against a use-case-specific passing threshold. Its methodologies cover LLM-as-judge (scalable, consistent evaluation via a prompt providing the task description, agent output, ground truth if available, a scale with level descriptions, and a request for structured judgment), human evaluation (catching hallucinations, system failures, and subtle biases automation misses, via systematic sampling and pattern tracking), and end-state evaluation (for agents that mutate persistent state, judging whether the final state matches expectations rather than the path taken).

Its test set design starts with small samples during development (where changes have dramatic, easily-visible impact), sampled from real usage patterns plus known edge cases, stratified across four complexity levels: simple (single tool call), medium (multiple tool calls), complex (many tool calls, significant ambiguity), and very complex (extended interaction, deep reasoning). Its context engineering evaluation section validates context strategies by running agents with different approaches on the same test set and comparing quality/token/efficiency metrics, plus degradation testing at different context sizes to find performance cliffs and establish safe operating limits. Its continuous evaluation section covers automated pipelines running on every agent change with version-over-version comparison, and production monitoring via sampled interaction evaluation, quality-drop alerts, and trend dashboards.

Its eight-step practical guidance for building evaluation frameworks: define relevant quality dimensions, create rubrics with clear actionable level descriptions, build test sets from real usage and edge cases, implement automated pipelines, establish baseline metrics before changes, run evaluations on all significant changes, track metrics over time, and supplement automation with human review. Its five pitfalls to avoid: overfitting to specific paths (evaluate outcomes, not steps), ignoring edge cases, single-metric obsession (use multi-dimensional rubrics), neglecting context effects (test with realistic context sizes), and skipping human evaluation (automation misses subtle issues). It connects to all other skills in its collection as a cross-cutting concern for validating their effectiveness.

test_set = [
    {
        "name": "simple_lookup",
        "input": "What is the capital of France?",
        "expected": {"type": "fact", "answer": "Paris"},
        "complexity": "simple",
        "description": "Single tool call, factual lookup"
    },
    {
        "name": "medium_query",
        "input": "Compare the revenue of Apple and Microsoft last quarter",
        "complexity": "medium",
        "description": "Multiple tool calls, comparison logic"
    },
    {
        "name": "research_synthesis",
        "input": "Research emerging AI technologies, evaluate their potential impact, and recommend adoption strategy",
        "complexity": "very_complex",
        "description": "Extended interaction, deep reasoning, synthesis"
    }
]

Use this skill when testing agent performance systematically, validating context engineering choices, measuring improvements over time, catching regressions before deployment, building quality gates for agent pipelines, comparing agent configurations, or evaluating production systems continuously.

Inputs: an agent system and a set of test queries spanning complexity levels, plus the quality dimensions relevant to the use case.

Outputs: a multi-dimensional rubric with weighted scores, an LLM-as-judge and/or human evaluation pipeline, a complexity-stratified test set, and continuous monitoring with trend tracking and regression alerts.

Connects as a cross-cutting concern to all other skills in the same collection, validating context engineering and agent architecture choices.

Agent-system engineers who need to systematically evaluate agent performance, validate context engineering choices, and catch regressions through multi-dimensional, complexity-stratified evaluation.