Compress frontier MoE models to sub-4-bit with validated performance

InstinctRazor quantizes any Hugging Face model to sub-4-bit precision and then recovers the resulting quality loss with on-policy distillation, OPD, distilling the original BF16 teacher's completed generation trajectories into a per-expert LoRA on the quantized student, re-quantizing afterward so the footprint stays unchanged. Supporting a new model family requires writing one ModelAdapter, not forking the whole pipeline; per-model code is keyed on the model's config.model_type, and coverage is validated against transformers 5.9 in tests/test_adapter_coverage.py.

Its default quantization recipe uses INT3 for expert weights and INT4 for the backbone, in groups of 128 with symmetric clip-search, no calibration data needed, while protecting the router, norms, and vision components in BF16, yielding an effective bit-width of about 3.05. It also supports comparing against AWQ, GPTQ, RTN, and ParoQuant at matched bit-widths, deploying either as a llama.cpp GGUF file or a dequantized-BF16 checkpoint for clean evaluation, and evaluating against MMLU-Pro, GPQA, MMMLU, MATH-500, AIME, LiveCodeBench, HumanEval, MBPP, HLE, and multimodal benchmarks. It handles fused-expert MoE, separate-expert MoE, and dense model layouts with shape-agnostic quantization math; quantization is universal since transformers 5.x already batches every MoE family into fused expert tensors, but OPD recovery needs the model's experts to share a common reference forward pass, gate_up_proj and down_proj, chunk(2), routing-in-parent, validated on qwen3_moe, while families with different expert math, such as gpt_oss's bias-plus-clamp-plus-interleaved-GLU, need a small per-family expert-compute hook for OPD, quantization itself still works universally.

./razor --model <any-hf-model> --quant instinct-iq3 --recover opd --eval mmlu_pro,gpqa,math500

Two worked results: quantizing Qwen3.5-122B-A10B produces a 47 GB GGUF that runs on a single 80 GB GPU, and the shipped 122B IQ3_XXS artifact, 48 GiB, scores MMLU-Pro 90.7 and GPQA 80.8 at 115.9 tokens per second on one H100, 45.7 tokens per second with expert offload, beating a footprint-matched Gemma-4-26B-A4B on MMLU-Pro, 88.5 vs 85.6, GPQA-Diamond, 84.8 vs 79.3, and MMMU-Pro, 80.8 vs 73.8, though the BF16 teacher and Gemma both edge it out on LiveCodeBench v6. Applying OPD recovery to a 3-bit Qwen3.6-35B-A3B student raises MATH-500 accuracy from 81.7 to 89.2, versus a 90.0 BF16 teacher, and GPQA accuracy from 68.7 to 77.3, versus 84.3 for the teacher, with most of the gap traced to answer truncation, which OPD sharply reduces, from 19 out of 120 to 1 out of 120 truncated on MATH-500 and 64 out of 198 to 30 out of 198 on GPQA.

Use this when you need to shrink a Hugging Face model below 4 bits per weight while recovering most of the resulting quality loss, especially for MoE models where per-expert quantization and OPD apply cleanly. It is not needed if a model only requires standard 4-bit-or-above quantization, AWQ, GPTQ, and RTN are supported as comparison baselines, not replaced, and OPD recovery specifically only works out of the box on fused-MoE families sharing the standard expert forward pass, atypical expert math needs a custom hook first.

Input is any Hugging Face model identifier plus a quantization recipe, instinct-iq3, a GGUF type like Q4_K_M, or a dequant recipe like clip, awq, gptq, or rtn, and, for recovery, a --recover opd flag with a --student checkpoint. Output is a quantized GGUF file or a dequantized-BF16 evaluation checkpoint, optionally recovered via OPD, plus evaluation scores on the requested benchmark set.

Hugging Face Transformers, 5.x, adapter keyed on config.model_type, llama.cpp for writing and serving GGUF files, and a separate training virtual environment with a different torch build specifically for OPD recovery.

ML engineers who need to deploy large Hugging Face models, especially large MoE models, at sub-4-bit precision on limited GPU memory while recovering most of the accuracy lost to quantization.