Evaluate LLM outputs with production-grade judge systems

Advanced Evaluation is a skill covering production-grade techniques for evaluating LLM outputs with LLMs as judges. It synthesizes academic research, industry practice and implementation experience into patterns for building reliable evaluation systems. Its key insight is that LLM-as-a-Judge is a family of approaches, each suited to different contexts, and that choosing the right one and mitigating known biases is the core competency it develops.

It distinguishes two primary approaches. Direct scoring has a single LLM rate one response on a defined scale. It is best for objective criteria such as factual accuracy, instruction following and toxicity, with moderate to high reliability, and its failure modes are score calibration drift and inconsistent scale interpretation. Pairwise comparison has an LLM compare two responses and pick the better. It is best for subjective preferences such as tone, style and persuasiveness, is more reliable than direct scoring for preferences, and fails through position bias and length bias. The skill cites the MT-Bench paper by Zheng and colleagues, which found pairwise comparison agrees more with human judges for preference-based evaluation.

Activate it when building automated evaluation pipelines, comparing model responses to select the best, setting consistent quality standards across evaluation teams, debugging evaluation systems with inconsistent results, designing A/B tests for prompt or model changes, creating rubrics, or analyzing correlation between automated and human judgments. It is also triggered by phrases such as implement LLM-as-judge, compare model outputs, create evaluation rubrics and mitigate evaluation bias.

Its own limitations note applies: use it when the task clearly matches this scope, and do not treat its output as a substitute for environment-specific validation, testing or expert review. The skill stresses validating automated judgments against human ones, since automated evaluation is only valuable if it correlates with human assessment.

The skill names five judge biases and a mitigation for each. Position bias favors the first response, so evaluate twice with swapped positions and use a consistency check. Length bias rates longer responses higher, so prompt explicitly to ignore length or use length-normalized scoring. Self-enhancement bias makes models rate their own outputs higher, so use different models for generation and evaluation. Verbosity bias rewards unnecessary detail, so use rubrics that penalize irrelevant detail. Authority bias rewards a confident tone, so require evidence citation and a fact-checking layer.

Metrics depend on task type: recall, precision and F1 for binary pass or fail with Cohen's kappa as secondary; Spearman's rho and Kendall's tau for ordinal ratings; agreement rate and position consistency for pairwise preferences; and macro and micro F1 for multi-label tasks. The skill notes that systematic disagreement patterns matter more than high absolute agreement.

A direct scoring setup needs clear criteria, a calibrated scale and structured output, and each criterion is written in a fixed pattern:

Criterion: [Name]
Description: [What this criterion measures]
Weight: [Relative importance, 0-1]

Scales of 1 to 3, 1 to 5 and 1 to 10 are compared, with 1 to 10 advised only alongside detailed rubrics. Every scoring prompt must require justification before the score, which it says improves reliability by 15 to 25 percent over score-first prompts.

Pairwise comparison uses a position-swap protocol: run once with A first, again with B first, and if the passes disagree return a tie with confidence 0.5; if they agree, use the winner with averaged confidence. The prompt instructs the judge not to prefer longer responses or a particular position and to accept ties.

Rubric generation covers level descriptions, observable characteristics, examples, edge cases and scoring guidelines, with lenient, balanced and strict calibration and domain-specific terminology. It says well-defined rubrics reduce evaluation variance by 40 to 60 percent.

A pipeline has a criteria loader, a primary scorer (direct or pairwise), bias mitigation, and confidence scoring, ending in scores, justifications and confidence. A decision tree says to use direct scoring where there is objective ground truth, pairwise for preference or quality judgments, and reference-based evaluation for tasks like summarization or translation. For scale, it suggests a panel of LLMs, hierarchical evaluation with a cheap screening model and an expensive model for edge cases, and human review for low-confidence cases. Named anti-patterns include scoring without justification, single-pass pairwise comparison, overloaded criteria, missing edge case guidance and ignoring confidence calibration. Three worked examples show direct scoring for accuracy, a pairwise comparison with position swap, and rubric generation for code readability.

It integrates with the context-fundamentals, tool-design, context-optimization and foundational evaluation skills, and references work by Eugene Yan, the G-Eval paper and papers on LLM evaluator fairness.

ML engineers, prompt engineers and evaluation teams who need trustworthy, bias-aware automated evaluation of LLM outputs.