Evaluate LLMs Locally with Transformers.js

This Promptfoo config runs a fully local LLM evaluation using Transformers.js, with no calls to a hosted API. The generation provider is onnx-community/Qwen3-0.6B-ONNX, Qwen3's smallest model at 0.6B parameters with thinking capabilities, configured for sampling (temperature 0.6, top-p 0.95, up to 100 new tokens); the model downloads once, around 600MB, and is cached locally afterward. Embeddings for the similar semantic-similarity assertions also run locally, via Xenova/all-MiniLM-L6-v2 with mean pooling and normalization - so both generation and grading run without any external API calls.

Three test cases ask the model to explain a beginner concept - machine learning, photosynthesis, gravity - in 2-3 sentences, each checked against a reference explanation via similar (thresholds of 0.5, 0.5, and 0.4 respectively) plus a minimum output-length check requiring over 50 characters.

Use it as a template for evaluating a small local model fully offline, with local embeddings for similarity grading, when API cost, latency, or data-privacy constraints rule out hosted providers. Do not use it if you need a larger, more capable model than a 0.6B local model can provide, or if you're comfortable with hosted API calls.

Input: the YAML config - a beginner-explanation prompt template, the local generation and embedding provider configs, and three concept test cases. Output: Promptfoo's evaluation report, scoring each explanation's semantic similarity to a reference answer and its minimum length, computed entirely locally.

Uses Promptfoo's transformers provider, via Transformers.js, for both text generation (onnx-community/Qwen3-0.6B-ONNX) and feature extraction and embeddings (Xenova/all-MiniLM-L6-v2), with no external API dependency.

Teams that need to evaluate a model entirely offline - for cost, latency, or data-privacy reasons - including similarity-based grading without calling an external embedding API.