What models does this local LLM evaluation setup use?

It uses `onnx-community/Qwen3-0.6B-ONNX` (~600MB) for text generation with thinking capabilities, and `Xenova/all-MiniLM-L6-v2` (~23MB) for local embeddings and similarity assertions.

Do I need API keys to run this?

No, this is a completely local setup that requires no API keys or external services — it runs entirely offline after the initial model download.

How do I get started with this example?

Install the dependency with `npm install @huggingface/transformers`, then run `npx promptfoo@latest init --example provider-transformers-local`, navigate to the directory, and execute `npx promptfoo@latest eval`. The first run downloads both models; subsequent runs use the cache and are much faster.

When should I use this instead of frontier hosted models?

Use this when you need offline evaluation, cost-free testing, or privacy-sensitive data that shouldn't leave your machine — not when you need to compare against top-tier output quality, since the local models are small (0.6B parameters) chosen for feasibility on ordinary hardware.

Can I speed up inference on this setup?

Yes, you can use `device: webgpu` for faster inference if supported, `dtype: q4` to reduce memory footprint via quantized models, or `-j 1` to help on systems with limited RAM.

Prompt Chain

Evaluate LLMs Locally with Transformers.js

Name: Provider Transformers Local
Availability: OnlineOnly
Author: Promptfoo

Promptfoo example running a fully local, offline LLM eval with Transformers.js - no API keys required.

Copy chain

Works with transformers.js

Promptfoo

Own this? Claim it

Spark score

out of 100

Updated last month

Version code-scan-action-0.1

Models

qwen 2 5

Add to Favorites

Why it matters

Set up a completely local LLM evaluation environment using Transformers.js. This asset requires no API keys or external services, enabling private and secure model testing.

Outcomes

What it gets done

Demonstrate local LLM evaluation

Integrate with Transformers.js

Eliminate need for API keys and external services

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/pfoo-provider-transformers-local | bash

Steps

Steps in the chain

Install Transformers.js dependency

Initialize example project

Navigate to project directory

Run evaluation

Overview

Provider Transformers Local

A promptfoo example running a fully local, offline LLM evaluation with Transformers.js, using small ONNX models for generation and embeddings. Use when you need offline, API-key-free, or privacy-sensitive LLM evaluation and can accept smaller local models.

What it does

This example demonstrates a completely local LLM evaluation setup using Transformers.js - no API keys or external services required. It shows local text generation with onnx-community/Qwen3-0.6B-ONNX (the latest Qwen3 model with thinking capabilities, ~~600MB), local embeddings with Xenova/all-MiniLM-L6-v2 for similarity assertions (~~23MB), and fully offline evaluation after the initial model download.

When to use - and when NOT to

Use this example when you need to run evals with no external API dependency - offline environments, cost-free testing, or privacy-sensitive data that shouldn't leave the machine. It is not intended for comparing against frontier hosted models - the local models here are small (0.6B parameters) and chosen for feasibility on ordinary hardware, not top-tier output quality.

Inputs and outputs

Install the optional dependency and run:

npm install @huggingface/transformers
npx promptfoo@latest init --example provider-transformers-local
cd provider-transformers-local
npx promptfoo@latest eval

The first run downloads both models (Qwen3-0.6B-ONNX ~600MB, all-MiniLM-L6-v2 ~23MB); subsequent runs use the cache and are much faster. Providers are configured as transformers:text-generation:onnx-community/Qwen3-0.6B-ONNX (with maxNewTokens, temperature, topP, doSample) and, for embeddings, transformers:feature-extraction:Xenova/all-MiniLM-L6-v2 set as defaultTest.options.provider.embedding.

Integrations

Integrates Transformers.js as a fully local provider backend, running entirely on CPU by default. For faster inference, device: webgpu can be used if supported; dtype: q4 reduces memory footprint via quantized models; and -j 1 helps on systems with limited RAM.

Who it's for

Teams needing offline, API-key-free, or privacy-sensitive LLM evaluation who accept smaller local models in exchange for zero external dependency and zero per-request cost.

Source README

provider-transformers-local (Fully Local LLM Evaluation)

This example demonstrates a completely local LLM evaluation setup using Transformers.js - no API keys or external services required.

Prerequisites

Install the optional Transformers.js dependency:

npm install @huggingface/transformers

Usage

npx promptfoo@latest init --example provider-transformers-local
cd provider-transformers-local
npx promptfoo@latest eval

What This Example Shows

Local text generation with onnx-community/Qwen3-0.6B-ONNX (latest Qwen3 model with thinking capabilities)
Local embeddings with Xenova/all-MiniLM-L6-v2 for similarity assertions
Fully offline evaluation after initial model download
No API keys needed

Models Used

Model	Task	Size	Purpose
`onnx-community/Qwen3-0.6B-ONNX`	Text Generation	~600MB	Generate responses
`Xenova/all-MiniLM-L6-v2`	Embeddings	~23MB	Similarity assertions

First Run

The first evaluation downloads both models (cached for subsequent runs):

Downloading Qwen3-0.6B-ONNX... ~600MB
Downloading all-MiniLM-L6-v2... ~23MB

Subsequent runs use cached models and are much faster.

Configuration Highlights

providers:
  - id: transformers:text-generation:onnx-community/Qwen3-0.6B-ONNX
    config:
      maxNewTokens: 100
      temperature: 0.6
      topP: 0.95
      doSample: true

defaultTest:
  options:
    provider:
      embedding:
        id: transformers:feature-extraction:Xenova/all-MiniLM-L6-v2

Notes

Runs entirely on CPU by default
For faster inference, use device: webgpu if your system supports it
Use dtype: q4 for smaller memory footprint with quantized models
Run with -j 1 for systems with limited RAM

FAQ

Common questions

Discussion