Design & Implement LLM Evaluation Frameworks

This skill designs, implements, and optimizes comprehensive evaluation frameworks for large language models across multiple dimensions: capability (task-specific performance on QA, summarization, reasoning), safety (harmful content detection, bias assessment, robustness), alignment (human preference alignment, instruction following), and efficiency (latency, throughput, computational cost). Its design principles favor stratified sampling for balanced test sets, combining automated metrics with human evaluation protocols, real-world usage patterns in evaluation tasks, and explicit adversarial and edge-case testing.

For automated metrics it implements an evaluator that scores generation quality with ROUGE-1/2/L and BERTScore for semantic similarity, and classification tasks with accuracy and macro/weighted F1:

import numpy as np
from sklearn.metrics import accuracy_score, f1_score
from rouge_score import rouge_scorer
from bert_score import score as bert_score

class LLMEvaluator:
    def __init__(self):
        self.rouge_scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)
    
    def evaluate_generation(self, predictions, references):
        metrics = {}
        
        # ROUGE scores for summarization/generation
        rouge_scores = [self.rouge_scorer.score(ref, pred) for pred, ref in zip(predictions, references)]
        metrics['rouge1'] = np.mean([score['rouge1'].fmeasure for score in rouge_scores])
        metrics['rouge2'] = np.mean([score['rouge2'].fmeasure for score in rouge_scores])
        metrics['rougeL'] = np.mean([score['rougeL'].fmeasure for score in rouge_scores])
        
        # BERTScore for semantic similarity
        P, R, F1 = bert_score(predictions, references, lang='en', verbose=False)
        metrics['bert_score'] = F1.mean().item()
        
        return metrics

A custom evaluation pipeline batches predictions through a model/tokenizer and routes them to per-task evaluators (generation, safety, reasoning) before aggregating results. Benchmark integration wraps standard suites - HellaSwag, MMLU, HumanEval, TruthfulQA - each with its own data loader, prompt formatter, and scoring function. Human evaluation defines rated criteria (helpfulness, harmlessness, honesty, coherence, each 1-5), builds multi-annotator comparison tasks, and computes inter-annotator agreement via pairwise Cohen's kappa. Safety and bias evaluation scores toxicity (mean, max, and rate above a threshold) and tests bias across demographic groups (gender, race, religion) by templating prompts per demographic and measuring sentiment mean, variance, and a fairness score. Reporting generates a structured report with summary metrics, detailed results, visualizations (including a radar chart across evaluation categories), and recommendations.

Use it when you need a rigorous, multi-dimensional evaluation of an LLM - not just one accuracy number - spanning capability, safety, alignment, and efficiency, with both automated and human-annotated evidence. Its own best practices flag what NOT to do: don't rely on a single metric without considering task-specific requirements, don't skip human evaluation for subjective quality judgments, don't evaluate with insufficient sample sizes for reliable statistical conclusions, and watch for data leakage between training and evaluation sets and for demographic bias baked into the evaluation datasets themselves.

Input is the model under test, its tokenizer, and labeled or reference test datasets per task. Output is per-task metric dictionaries, benchmark scores, human-annotation agreement statistics, safety/bias evaluation results, and a consolidated report with visualizations and recommendations.

It combines scikit-learn (accuracy, F1, Cohen's kappa), rouge-score and bert-score for generation quality, HuggingFace datasets for benchmark loading (HellaSwag, MMLU, HumanEval, TruthfulQA), and matplotlib/seaborn for visualization.

ML engineers and researchers building or auditing an LLM evaluation pipeline who need automated metrics, standard benchmark scoring, human evaluation with agreement tracking, and safety/bias testing in one framework.