Run local GPU evaluations for Hugging Face models

Hugging Face Community Evals runs evaluations of Hugging Face Hub models on local hardware, using inspect-ai and lighteval. It helps choose between the vllm, Hugging Face Transformers and accelerate backends, and covers smoke tests, task selection and backend fallback strategy. It is an official Hugging Face skill under the Apache-2.0 license, and local execution is done with uv run.

Three bundled scripts do the work:

• scripts/inspect_eval_uv.py: a local inspect-ai eval on a Hub model through Hugging Face Inference Providers, the path with the lowest setup overhead. It fixes the connection count at 1 and uses a small positive temperature of 0.001 because the providers do not accept temperature 0, and it defaults to a local sandbox.
• scripts/inspect_vllm_uv.py: a local GPU eval with inspect-ai using vllm (the default backend, with temperature 0.0, 4 connections, --tensor-parallel-size and --gpu-memory-utilization defaulting to 0.8) or Transformers via --backend hf (1 connection, temperature 0.001, --device and --dtype options).
• scripts/lighteval_vllm_uv.py: a local GPU eval with lighteval using vllm or accelerate, with --tasks, --max-samples, --batch-size (default 1), --output-dir, --use-chat-template and --system-prompt for chat models.

The core workflow has five steps: choose the framework (inspect-ai for explicit task control and inspect-native flows, lighteval when the benchmark is naturally a lighteval task string, especially leaderboard-style tasks), choose the backend (prefer vllm for throughput on supported architectures, and use Transformers or accelerate as compatibility fallbacks), start with a smoke test (--limit 10 for inspect-ai or --max-samples 10 for lighteval), scale up only after the smoke test passes, and hand off to remote execution if the user wants it.

Task selection: inspect-ai examples are mmlu, gsm8k, hellaswag, arc_challenge, truthfulqa, winogrande and humaneval. lighteval task strings use suite|task|num_fewshot, for example leaderboard|mmlu|5, leaderboard|gsm8k|5, leaderboard|arc_challenge|25 and lighteval|hellaswag|0, and several tasks can be comma-separated in --tasks.

Use it to run evaluations against Hub models on local hardware: backend selection, local GPU evals, and choosing between vLLM, Transformers and accelerate.

It does not cover Hugging Face Jobs orchestration, model-card or model-index edits, README table extraction, Artificial Analysis imports, .eval_results generation or publishing, or PR creation and community-evals automation. To run the same eval remotely, hand off to the hugging-face-jobs skill and pass it one of these scripts with the chosen arguments, which also covers picking remote hardware, passing secrets, scheduling recurring runs and monitoring jobs. To publish results into the community evals workflow, stop after generating the run and hand off that step. Use it only when the task matches its upstream scope, and verify commands, API behavior, pricing, quotas and credentials against current documentation. Generated examples do not replace environment-specific tests, security review or approval for costly actions.

Prerequisites are uv and, for gated or private models, an HF_TOKEN. For local GPU runs, verify GPU access with nvidia-smi first; if it is unavailable, use inspect_eval_uv.py for lighter provider-backed evaluation or hand off to Jobs. A quick smoke test looks like this:

uv run scripts/inspect_eval_uv.py \
  --model meta-llama/Llama-3.2-1B \
  --task mmlu \
  --limit 20

The GPU variants run inspect_vllm_uv.py with a model and task (for example meta-llama/Llama-3.2-1B on gsm8k), add --backend hf --trust-remote-code for an architecture such as microsoft/phi-2 that vLLM does not support, and run lighteval with --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" --use-chat-template or --backend accelerate. Outputs are evaluation runs and logs from the chosen framework, with results in an output directory for lighteval.

Hardware guidance: models under 3B fit consumer GPUs, Apple Silicon or a small dev GPU; 3B to 13B want a stronger local GPU; and 13B or larger need a high-memory GPU or a hand-off to Jobs. For smoke tests, prefer cheaper local runs with a sample limit. Troubleshooting: for CUDA or vLLM out-of-memory errors, reduce --batch-size or --gpu-memory-utilization, switch to a smaller model, or hand off to Jobs; if vLLM does not support the model, use --backend hf for inspect-ai or --backend accelerate for lighteval; if gated repo access fails, verify HF_TOKEN; and if the model needs custom code, add --trust-remote-code.

It works with inspect-ai, inspect-evals, lighteval, vLLM, Hugging Face Transformers, accelerate, PyTorch, uv, Hugging Face Inference Providers and the hugging-face-jobs skill.

It suits ML engineers and researchers who want to benchmark Hub models locally with the right framework and backend before scaling out.