Run local GPU evaluations for Hugging Face models

This skill runs evaluations against Hugging Face Hub models on local hardware, covering inspect-ai with local inference, lighteval with local inference, and choosing between vllm, Hugging Face Transformers, and accelerate as the inference backend. It ships three scripts for three distinct paths: scripts/inspect_eval_uv.py for a local inspect-ai eval via Hugging Face Inference Providers (lowest setup overhead, no direct GPU control needed), scripts/inspect_vllm_uv.py for a local GPU inspect-ai eval using vllm or a Transformers fallback (--backend hf --trust-remote-code) for unsupported architectures, and scripts/lighteval_vllm_uv.py for a local GPU lighteval run using vllm or an accelerate fallback, particularly suited to Open LLM Leaderboard-style task strings (suite|task|num_fewshot, e.g. leaderboard|mmlu|5). The core workflow is to pick the framework (inspect-ai for explicit task control, lighteval for leaderboard-style task strings), pick the backend (vllm for throughput, Transformers/accelerate as compatibility fallbacks), always start with a smoke test (--limit 10 for inspect-ai, --max-samples 10 for lighteval) before scaling up, and hand off to a separate hugging-face-jobs skill for remote execution using the same script and arguments. It documents named eval tasks for each framework (mmlu, gsm8k, hellaswag, arc_challenge, truthfulqa, winogrande, humaneval for inspect-ai; leaderboard/lighteval task strings for lighteval), a hardware-sizing table (under 3B params on a consumer GPU/Apple Silicon, 3B-13B on a stronger local GPU, 13B+ on high-memory hardware or handed off remotely), and troubleshooting steps for CUDA/vLLM out-of-memory errors, models unsupported by vLLM, gated-repo access failures, and custom model code requiring --trust-remote-code.

Use it specifically for running evals against Hub models on local hardware - backend selection, local GPU evals, and choosing between vLLM/Transformers/accelerate. It explicitly does NOT cover Hugging Face Jobs orchestration (remote hardware, secrets, scheduling, job monitoring - hand off to the hugging-face-jobs skill instead), model-card or model-index edits, README table extraction, Artificial Analysis imports, .eval_results generation/publishing, or PR creation for the community-evals workflow. If the user wants results published into the community-evals workflow, this skill's job ends after generating the evaluation run - publishing is handed off to ~/code/community-evals.

Inputs: a Hub model ID, a task or task string, an inference backend choice, and an HF_TOKEN for gated/private models. Outputs: local evaluation results from inspect-ai or lighteval runs, verified first via a smoke test (--limit/--max-samples), scaled up only after that passes, or a hand-off package (script + args) for remote execution via hugging-face-jobs.

uv run scripts/inspect_eval_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --task mmlu \
  --limit 20

Built on uv run for local execution, inspect-ai and lighteval as the two evaluation frameworks, and vllm/Hugging Face Transformers/accelerate as inference backends. Requires HF_TOKEN for gated or private models and nvidia-smi to confirm GPU access before local GPU runs. Hands off to the separate hugging-face-jobs skill for remote execution and to ~/code/community-evals for publishing results.

ML engineers and researchers who need to evaluate a Hugging Face Hub model locally before committing to remote compute or publishing results to the community-evals workflow.