Evaluate LLM Performance Systematically

This skill covers comprehensive evaluation strategies for LLM applications, from automated metrics to human evaluation and A/B testing. It defines three evaluation types: automated metrics (text generation - BLEU, ROUGE, METEOR, BERTScore, perplexity; classification - accuracy, precision/recall/F1, confusion matrix, AUC-ROC; retrieval/RAG - MRR, NDCG, Precision@K, Recall@K), human evaluation across six dimensions (accuracy, coherence, relevance, fluency, safety, helpfulness), and LLM-as-Judge in four approaches (pointwise scoring, pairwise comparison, reference-based, reference-free). It provides an EvaluationSuite/Metric quick-start example, implementations for BLEU (nltk.translate.bleu_score), ROUGE (rouge_score.RougeScorer), and BERTScore (bert_score), and custom metrics for groundedness (NLI entailment check), toxicity (Detoxify), and factuality verification against a knowledge base. Its LLM-as-Judge patterns include a single-output quality rating prompt (accuracy/helpfulness/clarity scored 1-10 with JSON output) and a pairwise comparison prompt picking a winner with a confidence score. Its human-evaluation framework defines an AnnotationTask with 1-5 ratings for accuracy/relevance/coherence and issue flags (factual error, hallucination, off-topic, unsafe content), plus inter-rater agreement via Cohen's kappa. Its ABTest class runs a t-test and Cohen's d effect size to determine statistical significance and relative improvement between two variants. Its RegressionDetector flags any metric whose relative change drops below a configurable threshold against a baseline, and its BenchmarkRunner aggregates mean/std/min/max per metric across a benchmark dataset.

Use it for measuring LLM application performance systematically, comparing different models or prompts, detecting performance regressions before deployment, validating improvements from prompt changes, building confidence in production systems, establishing baselines, or debugging unexpected model behavior. It does not apply outside LLM evaluation - a different domain or tool is out of scope.

Inputs: test cases with expected outputs/context, a baseline result set for regression comparison, or two variants' scores for A/B testing. Outputs: computed metric scores (BLEU, ROUGE, BERTScore, custom metrics), LLM-judge ratings or pairwise verdicts, human-annotation results with inter-rater agreement, A/B test statistical significance and effect size, and a regression report flagging metrics that dropped below threshold.

nltk (BLEU), rouge_score, bert_score, transformers (NLI groundedness checks), detoxify (toxicity), scikit-learn (cohen_kappa_score), scipy.stats (t-test) and numpy (A/B test statistics), and an LLM API for LLM-as-Judge prompts.

from rouge_score import rouge_scorer

def calculate_rouge(reference, hypothesis):
    """Calculate ROUGE scores."""
    scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
    scores = scorer.score(reference, hypothesis)

    return {
        'rouge1': scores['rouge1'].fmeasure,
        'rouge2': scores['rouge2'].fmeasure,
        'rougeL': scores['rougeL'].fmeasure
    }

ML and applied-AI engineers who need to measure LLM application quality systematically - automated metrics, LLM-as-judge, human evaluation, A/B testing, and regression detection - rather than eyeballing outputs before deployment. Its best practices: use multiple, diverse metrics rather than one, test on representative real-world data, always compare against a baseline, apply proper statistical tests, integrate evaluation continuously into CI/CD, combine automated metrics with human validation, investigate failures through error analysis, and version-control evaluation results over time. Its common pitfalls: optimizing for a single metric at the expense of others, drawing conclusions from too small a sample, testing on contaminated training data, ignoring statistical variance, and using metrics misaligned with business goals.