Qwen3.8-Flash-Next (125B MoE) in NVFP4 on two 16 GB consumer GPUs.

Against Strata 0.1.44.1 on the same machine (2x RTX 5070 Ti 16 GB, Ryzen 9 9900X, 96 GB DDR5):

	capy-flash, NVFP4	Strata, same NVFP4 weights	Strata, 3-bit IQ3_S
prefill, 15k-token prompt	3,778 tok/s	2,703 tok/s	2,906 tok/s
prefill, 55k-token prompt	3,977 tok/s ¹	3,821 tok/s	4,037 tok/s
time to first token, agentic replay	1.82 s	2.42 s	1.86 s
generation, agentic replay	100.8 tok/s	78.3 tok/s	124.1 tok/s
tokens per agentic task run ²	42.9k	-	52.7k
time for 20 agentic task runs ²	4.28 h	-	4.32 h

• NVIDIA's official NVFP4 experts, used as published: not re-quantized.
• Fewer tokens, same finish time: Strata's 3-bit experts generate faster, but on the same coding tasks they
  write 23 % more tokens, so the agent finishes in the same time.
• Built for agents: 20/20 multi-step coding tasks, 0 malformed tool calls in 1,124; OpenAI- and
  Anthropic-compatible API.
• One engine, two models: it also serves the dense Qwen3.8-27B, exactly as ninfer-tp2 v0.4.9 did.

¹ Same prompts as the Strata runs, new text each time (the n-gram (PLE) table rows come from disk). Since v0.5.2 the
next chunk's rows are read while the GPUs run the current one; v0.5.3 keeps the rows' multipliers in RAM (the first
chunk of ~15k new tokens waits 0.3-0.5 s instead of 0.8-1.3 s) and computes the router on tensor cores.
² Strata 0.1.43 (0.1.44 did not change its IQ3_S path); 5 multi-step coding tasks x 4 runs, OpenCode. Details below.

Quick start (2x RTX 50-series 16 GB, an AVX-512 CPU, ~70 GiB of free RAM, 135 GB of disk):

curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash -s -- --check   # can this machine run it?
curl -fsSL https://raw.githubusercontent.com/ValerioDolci/capy-flash/main/install.sh | bash               # release + model into ~/capy-flash
~/capy-flash/current/serve-flash-next.sh ~/capy-flash/models/flash-next/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"capy-flash","messages":[{"role":"user","content":"What is 17*23?"}]}'                # from another shell

The installer checks GPUs, driver, AVX-512, RAM and disk and says what to fix, verifies every download, and prints
a test request; --model 27b installs Qwen3.8-27B instead, --dir picks another place. By hand: download a
release tarball and hf download Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash.

The routed experts that do not fit in VRAM live in host RAM and are computed on the CPU; everything else runs
tensor-parallel on the two GPUs. capy-flash is ninfer-tp2, renamed at v0.5.0: the two-GPU tensor-parallel fork
of Neroued/ninfer (v0.1-v0.4.9 under the old name); NOTICE lists every
piece of prior work it builds on.

Status: v0.5.4 (2026-10-16). Built, tested and measured on one machine (below). Not affiliated with Qwen or NVIDIA.

What it does

• Tensor parallel over PCIe on two GPUs without peer-to-peer (pinned-host mailbox, FP32 partial sums).
• A cache of routed experts in VRAM, partitioned by expert id between the two GPUs; missing experts are computed on
  the CPU (AVX-512 NVFP4 kernel with the GPU's W4A4 activation quantization), inside the captured CUDA graph.
• MTP speculative decoding (K = 3, draft vocabulary shortlist), INT8 KV with an elastic KV / expert-slot budget up to
  262,144 tokens, prefix cache across turns; OpenAI- and Anthropic-compatible server (ninfer-serve).
• The same binary still serves the dense Qwen3.8-27B (NVFP4, entirely on the two GPUs) exactly like ninfer-tp2 v0.4.9.
• Context cache for agents (v0.5.4): a request reserves its prompt plus a 4,096-token window of KV and grows it as the
  answer grows (a large max_tokens no longer pushes other conversations out of VRAM), and with two GPUs the cache can
  keep evicted conversations in RAM (--host-kv-mib, --host-state-slots; on in --profile 27b with 8 GiB, off on
  Flash-Next, whose experts already pin most of the RAM). On an agentic replay of Qwen3.8-27B (3 interleaved sessions
  and 11 subagents, production flags, 4 lanes): median time to first token 2.14 → 0.98 s, prompt tokens served from
  cache 67 → 78 %, prompt tokens recomputed −35 %, decode unchanged.

Measured

One machine: AMD Ryzen 9 9900X, 96 GB DDR5-5600, 2x RTX 5070 Ti 16 GB on PCIe 5.0 x8 + x8 (no P2P), Ubuntu, CUDA 13.1.
"GB300" below is the quality reference: NVIDIA's NVFP4 checkpoint served by vLLM on a GB300, same prompts, paired.

Quality (serving configuration of this README, v0.5.3, 2026-10-16):

benchmark	capy-flash	GB300 reference	verdict
GSM8K, 1,319 questions	0.9765	0.9810	not worse (McNemar p 0.070; 7 / 1 discordant)
MMLU-Pro, 1,008 questions	0.8452	0.8452	non-inferior (difference 0.00 pt, lower bound -1.37 pt, margin -2.5 pt)

v0.5.3 computes the router of prompt chunks on tensor cores, which sums in another order and can pick another expert
where two are tied; with the previous router (NINFER_QWEN4_ROUTER_MMA=0) the same gate gave 0.9780 / 0.8482 on the 2026-10-14 build.

Agentic replay (bench/capy_flash): 3 synthetic coding-agent sessions, 53 turns, prompts growing to
~81k tokens with ~90 % prefix reuse, 256 output tokens per turn, temperature 0. Same machine, same day:

engine	TTFT median (p90)	decode, server side	total
capy-flash	1.82 s (3.33)	100.8 tok/s	243 s
Strata 0.1.44.1, NVFP4 experts (the same weights)	2.42 s (5.03)	78.3 tok/s	314 s
Strata 0.1.44.1, GSQ-RCO IQ3_S experts (25 % smaller, lower quality)	1.86 s (4.53)	124.1 tok/s	236 s

Agent tasks: 5 multi-step coding tasks of a private task bank, 4 runs each, OpenCode with the model's sampling:
20/20 solved, 42.9k generated tokens per run (Strata 0.1.43 IQ3_S on the same tasks: 52.7k, 20/20; the GB300
reference: 26.5k), no malformed tool call in 1,124.

Prefill (same prompts on the three engines, server side, no prefix reuse): ~3.5k / ~15k / ~55k tokens:
capy-flash v0.5.3 1,985 / 3,778 / 3,977 tok/s (2026-10-16; v0.5.2: 1,856 / 3,302 / 3,715; v0.5.1: 1,941 / 3,435 /
2,791); Strata 0.1.44.1 IQ3_S 1,745 / 2,906 / 4,037; Strata NVFP4 1,347 / 2,703 / 3,821 (2026-10-15).

Start: a cold start of the Flash-Next server takes 26-29 s here (v0.5.2: 56-141 s), most of it reading the 64.6 GiB
of experts from disk.

Qwen3.8-27B (same machine, ninfer-tp2 v0.4.9 serving flags, agentic replay at temperature 0): the ninfer-tp2 gate
gives the same golden ids, perplexity (4k 4.385905, 64k 4.133481) and greedy texts (60/60) as v0.4.9; replay TTFT 1.42 s,
decode 171.2 tok/s, total 156 s (v0.4.9: 1.40 s, 172.2 tok/s, 154 s), the same token counts on all 53 turns.

Quality, prefill and start were measured on v0.5.3; the agentic replay and agent tasks on an internal build of 2026-10-14,
whose computation differs from v0.5.3 in one PLE table row (T155) and in the prompt router (T183, above); see the
development log, which explains every Tnn tag found in the sources.

Requirements

Linux, CUDA 13.1, two sm_120 GPUs (RTX 50 series) with 16 GB, a CPU with AVX-512 (F/BW/VL, VNNI), and about 70 GiB of
free RAM: the routed experts are one 64.6 GiB page-locked host allocation. The server checks this at startup and
refuses with the numbers when it does not fit (NINFER_RAM_GUARD=0 starts anyway).

Build

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DCMAKE_CUDA_ARCHITECTURES=120a \
  -DNINFER_BUILD_APPS=ON -DNINFER_BUILD_BENCHMARKS=OFF -DBUILD_TESTING=ON \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc
ninja -C build apps/ninfer-serve

Or take the prebuilt tarball of a release (Ubuntu, sm_120a):
it has serve-flash-next.sh and serve-tp2.sh (27B) with the profiles of this README (tools/release).

Model weights

Ready to serve: Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash
(126.6 GB, the artifact measured above, with its draft shortlist and SHA256SUMS; NVIDIA Open Model License together with
the Qwen Community License of Qwen3.8-Flash-Next):

hf download Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash --local-dir <dir>

Or convert nvidia/Qwen3.8-Flash-Next-NVFP4 yourself in one command (artifact + MTP draft shortlist, same result):

python tools/capy_flash/convert.py --model <checkpoint dir> --out <dir>

It runs python -m tools.convert --recipe qwen3_8_flash_next_nvfp4_idxbf16_gdnq8 --components text,mtp (experts NVFP4
as published, indexer BF16, GDN Q8, about 5 minutes on a GPU, 118 GB) and tools/qwen4_draft_shortlist.py.

Serve

build/apps/ninfer-serve <dir>/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer --profile flash-next --port 8098 --model-id capy-flash

--profile flash-next is the configuration measured above (--tp 2 --devices 0,1 --kv-dtype int8 --elastic --max-context 262144 --max-concurrency 1 --spec mtp --draft-tokens 3); --profile 27b is the ninfer-tp2 one for
Qwen3.8-27B (with Vision; --no-vision for artifacts without it). Flags given on the command line override the
profile's, in any order (ninfer-serve --help lists both). To keep it running: a systemd unit and a llama-swap entry
in examples/serving.

Docker. docker/Dockerfile.release packs a release tarball (1.2 GB, builds in about a
minute; the root Dockerfile compiles from source instead). It needs the NVIDIA Container Toolkit:

docker build -f docker/Dockerfile.release -t capy-flash .
docker run --gpus all -p 8080:8080 -v ~/capy-flash/models:/models capy-flash \
  /models/flash-next/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer --profile flash-next --host 0.0.0.0

Tested with rootless Podman and the two GPUs passed through (Flash-Next loads its 64.6 GiB of page-locked experts
without extra limits).
The CPU expert path quantizes activations like the GPU path (W4A4, on by default; NINFER_HOST_MOE_A4=0 restores
the older W4A8, which made the output depend on which experts happened to be in VRAM). --turn-anchors is 0 by
default on Flash-Next, which saves one prefill pass per agentic turn (about 0.8 s here); the cost is that a cancelled or
edited turn re-reads the whole conversation (--turn-anchors 1 restores the anchor). Models that sit entirely on the
GPU (Qwen3.8-27B) keep the anchor by default: there the extra pass costs nothing measurable. Other switches, diagnostics and dumps:
docs/capy-flash/diagnostics.md; general server options: docs/serving.md.

Tests

ctest --test-dir build runs the unit tests (187). The Engine integration tests run when their artifacts are set:
NINFER_TEST_QWEN4_ARTIFACT (a small synthetic Flash-Next artifact with the real layer geometry, made by
tools/capy_flash/make_test_artifact.sh <checkpoint dir> <out dir>, CPU only) and NINFER_TEST_ARTIFACT (a
Qwen3.8-27B artifact, for the tests inherited from ninfer-tp2). For the 27B, tools/tp2/gate.sh is the reference run:
it sets the artifacts, runs the two-device integration tests and excludes the seven single-GPU tests that need other
artifact kinds (DFlash, MoE, vision workspace; GATE_CTEST_EXCLUDE), then checks golden ids, perplexity and greedy
texts against a recorded reference (tools/tp2/README.md).

Known limits

• One request at a time (--max-concurrency 1); tested only on the machine above.
• With several requests decoding, an answer that outgrows its KV window when the GPUs are full first moves idle cached
  conversations to RAM (or drops them without a RAM tier); only if that cannot free a page does it end early with
  finish_reason: length (the request log counts it). --kv-lease-window 0 restores the full reservation of v0.5.3.
• Moving a conversation between RAM and the GPUs shares PCIe with the two GPUs' exchange: decode of other requests
  is slower while a move is in flight (no difference otherwise).
• Generated tokens per agent task are about 1.6x the GB300 reference (above): the remaining gap with the datacenter
  stack is length of reasoning, not correctness.
• Output is not bit-reproducible run to run: which experts sit in VRAM (GPU path) or not (CPU path) depends on timing,
  and long streamed prefills differ in the last bits; quality is measured with that in place.
• Decode is bound by host-RAM bandwidth for the experts computed on the CPU (about 270-300 per MTP window).

tools/release/README-release.md

capy-flash - prebuilt binaries

capy-flash (formerly ninfer-tp2) serves Qwen3.8-Flash-Next (MoE, NVFP4 experts, CPU offload) and Qwen3.8-27B
on two 16 GB Blackwell GeForce boards. Source and documentation: https://github.com/ValerioDolci/capy-flash.
COMMIT records the exact commit.

Contents

Path	What
bin/ninfer-serve	OpenAI- and Anthropic-compatible server
bin/ninfer	command-line generation
bin/ninfer-perplexity	perplexity evaluator
lib/libcudart.so.13	the CUDA runtime the binaries link (NVIDIA redistributable)
serve-flash-next.sh	launcher with the Qwen3.8-Flash-Next profile
serve-tp2.sh	launcher with the Qwen3.8-27B two-GPU production profile (as in ninfer-tp2)
SHA256SUMS	digests of every file above; check with sha256sum -c SHA256SUMS

Requirements

• Linux x86_64 with the distribution named in the tarball (the binaries link its FFmpeg and libcurl:
  libavcodec, libavformat, libavutil, libswscale, libcurl4). On another distribution
  build from source or use the Dockerfile.
• An NVIDIA driver that supports CUDA 13.1 (R580 or newer). No CUDA toolkit is needed.
• Two sm_120 boards with 16 GB for --tp 2 (validated on 2x RTX 5070 Ti without P2P).
• Qwen3.8-Flash-Next also needs a CPU with AVX-512 (F/BW/VL, VNNI) and about 70 GiB of free RAM: the routed
  experts that stay on the host are one 64.6 GiB page-locked allocation, and the server refuses to start without room
  for it (NINFER_RAM_GUARD=0 skips the check).

Run

tar -xzf capy-flash-*.tar.gz && cd capy-flash-*/
sha256sum -c SHA256SUMS
./serve-flash-next.sh /path/to/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer          # Qwen3.8-Flash-Next
./serve-tp2.sh /path/to/qwen3_8_27b_quasar_nvfp4.ninfer                           # Qwen3.8-27B
NINFER_PORT=8099 ./serve-flash-next.sh /path/to/model.ninfer --api-key secret      # extra flags are appended

Requests name the model capy-flash (Flash-Next) or qwen3.8-27b (27B); NINFER_MODEL_ID or --model-id changes it:

curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"capy-flash","messages":[{"role":"user","content":"What is 17*23?"}]}'

Artifacts:

• Qwen3.8-Flash-Next: Feyd89/Qwen3.8-Flash-Next-NVFP4-capy-flash (126.6 GB, NVIDIA's NVFP4 experts as published,
  SHA-256 in its SHA256SUMS), or convert nvidia/Qwen3.8-Flash-Next-NVFP4 yourself with tools/capy_flash/convert.py
  from the source tree (same result).
• Qwen3.8-27B: Feyd89/Qwen3.8-27B-QUASAR-QAT-nvfp4-NInfer (17.4 GB, SHA-256 in its SHA256SUMS) or the official
  Qwen3.8-27B-nvfp4-NInfer; both run with the same flags. For artifacts without a vision component set NINFER_VISION=0.

GET /health answers 200 while the Engine accepts work and 503 after an Engine-wide failure; the
server then exits with status 2, so run it under a supervisor that restarts on a non-zero exit
(Restart=on-failure, or llama-swap, which reloads the model on the next request).

tools/capy_flash/convert.py

#!/usr/bin/env python3
"""capy-flash: the v0.1 artifact in one command.

From the NVIDIA ModelOpt checkpoint (nvidia/Qwen3.8-Flash-Next-NVFP4) to the serving directory:
  1. tools.convert with the recipe qwen3_8_flash_next_nvfp4_idxbf16_gdnq8 (experts NVFP4 as published, indexer BF16,
     GDN Q8), components text + mtp  ->  <out>/qwen3_8_flash_next_nvfp4_idxgdnq8.ninfer
  2. the MTP draft shortlist beside it (tools/qwen4_draft_shortlist.py, --prefix 98304 --count-top 32768 over the
     repository ranking, plus the 8 192 most frequent Italian tokens it lacks: T186)  ->  <out>/qwen3_8_flash_next_nvfp4_idxgdnq8.draft_shortlist.txt
The engine picks the shortlist up by itself (artifact sidecar).

  python tools/capy_flash/convert.py --model <checkpoint dir> --out <dir> [--device cuda] [--skip-convert]

--skip-convert only (re)writes the shortlist next to an existing artifact. Run from the repository root.
"""
import argparse, subprocess, sys
from pathlib import Path

RECIPE = "qwen3_8_flash_next_nvfp4_idxbf16_gdnq8"
STEM = "qwen3_8_flash_next_nvfp4_idxgdnq8"
ROOT = Path(__file__).resolve().parents[2]

ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--model", type=Path, required=True, help="ModelOpt NVFP4 checkpoint directory (with tokenizer.json)")
ap.add_argument("--out", type=Path, required=True, help="output directory")
ap.add_argument("--device", default="cuda")
ap.add_argument("--name", default="qwen3.8-flash-next-idxgdnq8")
ap.add_argument("--skip-convert", action="store_true", help="only write the draft shortlist")
a = ap.parse_args()

if not (a.model / "tokenizer.json").is_file():
    sys.exit(f"capy_flash_convert: {a.model} has no tokenizer.json (not a checkpoint directory?)")
a.out.mkdir(parents=True, exist_ok=True)
art = a.out / f"{STEM}.ninfer"

def run(cmd):
    print("+", " ".join(str(c) for c in cmd), flush=True)
    subprocess.run([str(c) for c in cmd], cwd=ROOT, check=True)

if not a.skip_convert:
    run([sys.executable, "-m", "tools.convert", "--model", a.model, "--recipe", RECIPE, "--components", "text,mtp",
         "--device", a.device, "--name", a.name, "--out", art])
elif not art.exists():
    sys.exit(f"capy_flash_convert: --skip-convert but {art} does not exist")
run([sys.executable, ROOT / "tools/qwen4_draft_shortlist.py", "--prefix", "98304", "--count-top", "32768",
     "--extra-ranking", ROOT / "tools/freq_corpus/fixtures/ranking/it.train.counts.i64", "--extra-top", "8192",
     "--tokenizer", a.model, "--out", a.out / f"{STEM}.draft_shortlist.txt"])
print(f"capy_flash_convert: ready: {art} (+ draft shortlist)")

tools/qwen4_draft_shortlist.py

#!/usr/bin/env python3
"""T43: draft token shortlist for the Qwen4Exp MTP proposal head (EngineOptions draft_shortlist).

The shortlist is the union of
  * the token-ID prefix [0, --prefix): Qwen byte-level BPE assigns IDs in merge order, i.e. by
    pair frequency in the tokenizer's training corpus, so the low IDs are a frequency ranking of
    their own; the engine executes this block as a view of the output head (no copy);
  * the --count-top most frequent tokens of a measured ranking (default: the repository fixture
    tools/freq_corpus/fixtures/ranking/ranking.train.counts.i64 = teacher-forced argmax counts of
    Qwen3.6-27B over 72 464 chat conversations, 51.1 M emitted positions, en/zh/other chat, code,
    math; same 248 320-token vocabulary as Qwen3.8-Flash-Next, vocab.json identical); the engine
    copies these rows of the head (the ones above the prefix);
  * every added (special) token of the tokenizer below the public token count (chat control);
  * optionally (T186) the --extra-top most frequent tokens of a second ranking (--extra-ranking) not
    already in the list, e.g. tools/freq_corpus/fixtures/ranking/it.train.counts.i64 for Italian:
    with the default list 14.5 % of held-out Italian tokens fell outside it, 0.24 % with 8 192 more.
Reproducible: deterministic (stable sort on counts, ties by lower ID); the output header records
the inputs and the ranking file's SHA-256. Output: one ID per line, in priority order (specials,
the prefix ascending, then the ranking's tokens by count).

  qwen4_draft_shortlist.py --prefix 98304 --count-top 32768 \
      --tokenizer <NVIDIA Qwen3.8-Flash-Next-NVFP4 checkpoint dir> --out shortlist.txt
"""
import argparse, hashlib, json
from pathlib import Path
import numpy as np

VOCAB = 248320
HERE = Path(__file__).resolve().parent
DEFAULT_RANKING = HERE / "freq_corpus/fixtures/ranking/ranking.train.counts.i64"

ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--prefix", type=int, default=0, help="token-ID prefix [0, N)")
ap.add_argument("--count-top", type=int, default=0, help="most frequent N tokens of the ranking")
ap.add_argument("--ranking", type=Path, default=DEFAULT_RANKING)
ap.add_argument("--row", type=int, default=0, help="ranking row (0 = total; 1..5 = strata)")
ap.add_argument("--tokenizer", type=Path, required=True, help="directory with tokenizer.json")
ap.add_argument("--public", type=int, default=0, help="public token count (default: tokenizer size)")
ap.add_argument("--extra-ranking", type=Path, help="second ranking whose top tokens are appended")
ap.add_argument("--extra-top", type=int, default=0, help="new tokens taken from --extra-ranking")
ap.add_argument("--out", type=Path, required=True)
a = ap.parse_args()

tok = json.loads((a.tokenizer / "tokenizer.json").read_text())
base = len(tok["model"]["vocab"])
added = sorted(t["id"] for t in tok.get("added_tokens", []))
public = a.public or max([base - 1] + added) + 1
specials = [i for i in added if i < public]
raw = a.ranking.read_bytes()
rows = len(raw) // (VOCAB * 8)
counts = np.frombuffer(raw, dtype="<i8").reshape(rows, VOCAB)[a.row][:public]
order = np.argsort(-counts, kind="stable")
top = [int(i) for i in order[: a.count_top]] if a.count_top else []
seen, ids = set(), []
for i in specials + list(range(min(a.prefix, public))) + top:
    if i not in seen:
        seen.add(i)
        ids.append(i)
extra_note = ""
if a.extra_ranking and a.extra_top:
    extra_raw = a.extra_ranking.read_bytes()
    extra = np.frombuffer(extra_raw, dtype="<i8").reshape(-1, VOCAB)[0][:public]
    added_extra = 0
    for i in np.argsort(-extra, kind="stable"):
        if added_extra == a.extra_top or extra[i] <= 0:
            break
        if int(i) not in seen:
            seen.add(int(i))
            ids.append(int(i))
            added_extra += 1
    extra_note = (f", extra {added_extra} of {a.extra_ranking.name} "
                  f"(sha256 {hashlib.sha256(extra_raw).hexdigest()})")
sha = hashlib.sha256(raw).hexdigest()
with open(a.out, "w") as f:
    f.write(f"# T43 Qwen4Exp draft shortlist: {len(ids)} token IDs\n")
    f.write(f"# prefix {a.prefix}, count_top {a.count_top} of {a.ranking.name} row {a.row} "
            f"(sha256 {sha}){extra_note}, {len(specials)} added tokens, public {public}\n")
    f.write(f"# tokenizer {a.tokenizer} (vocab {base} + {len(added)} added)\n")
    for i in ids:
        f.write(f"{i}\n")
print(json.dumps({"out": str(a.out), "ids": len(ids), "prefix": a.prefix, "count_top": a.count_top,
                  "outside_prefix": sum(1 for i in ids if i >= a.prefix), "specials": len(specials),
                  "public": public, "ranking_sha256": sha}))

tools/capy_flash/make_test_artifact.sh

#!/bin/bash
# capy-flash: the small Qwen3.8-Flash-Next test artifact of the Engine integration tests
# (NINFER_TEST_QWEN4_ARTIFACT: ninfer_qwen4_tp2_engine_test, ninfer_qwen4_mtp_engine_test).
#
#   tools/capy_flash/make_test_artifact.sh <NVIDIA Qwen3.8-Flash-Next-NVFP4 checkpoint dir> <out dir>
#
# 1. gen_synthetic.py (sybrix, Apache-2.0): a random checkpoint with the REAL per-layer geometry
#    (hidden 2560, 4 hyper-connection streams, 512 NVFP4 experts top-10 + shared, GDN and QSA layers,
#    PLE at block 1, MTP head), only 4 Text layers (GDN GDN GDN QSA) and a tiny n-gram table; tokenizer
#    and config come from the real checkpoint (only those files are read). Seed 1234. ~11 GB, CPU only.
# 2. tools.convert --recipe qwen3_8_flash_next_nvfp4 --components text,mtp (CPU, under a minute).
# Prints the export line for the tests. Needs Python with torch, numpy and safetensors.
set -euo pipefail
[ $# -eq 2 ] || { sed -n 2,13p "$0"; exit 2; }
SRC=$1; OUT=$2; PY=${PYTHON:-python3}
ROOT=$(cd "$(dirname "$0")/../.." && pwd)
mkdir -p "$OUT"
[ -f "$OUT/fn4/model.safetensors.index.json" ] ||
  "$PY" "$ROOT/tools/capy_flash/gen_synthetic.py" --source "$SRC" --out "$OUT/fn4" --layers 4 --mtp --seed 1234
(cd "$ROOT" && "$PY" -m tools.convert --model "$OUT/fn4" --recipe qwen3_8_flash_next_nvfp4 --components text,mtp \
   --device cpu --name capy-flash-fn4 --out "$OUT/fn4.ninfer")
echo "export NINFER_TEST_QWEN4_ARTIFACT=$OUT/fn4.ninfer"

tools/tp2/gate.sh

#!/usr/bin/env bash
# Behavior-preservation gate for the two-GPU layer: one command, PASS/FAIL per stage.
#
#   tools/tp2/gate.sh [--record] [--stages LIST] [--attention] [--out DIR] <build-dir> <reference-dir>
#
# --record writes <reference-dir> from this build instead of comparing against it. Stages, in order
# (default: all but build):
#   build    cmake --build <build-dir> (no GPU)
#   shards   tools/tp2/check_shard_sources.py on the source tree (no GPU): the shards compiled from
#            upstream's sources (docs/maintainer/upstream-merge.md 2.4) cannot go silently wrong
#   ctest    the unit, Op and two-device tests that a tp2 change can reach, plus the tp2 *_real tests
#            (NINFER_TEST_ARTIFACT); --attention adds ninfer_softmax_attention_test (~12 min)
#   golden   tools/golden/record.sh on the synthetic tp1 model; ids must equal the reference's
#   ppl      ninfer-perplexity --tp 2 --quick on the QUASAR artifact, 65536/32768 and 4096/2048,
#            INT8 KV; every printed digit of the per-source table must equal the reference's.
#            GATE_PPL_PAIRED=1 (for commits meant to change bits): a table that differs is compared
#            window by window with the reference's report (tools/tp2/ppl_paired.py): PASS if not
#            worse at p 0.05 and |change| < 0.5 %, WARN otherwise
#   greedy   ninfer-serve with the production flags (MTP3, --lm-head-draft, C=1), the reference's
#            60 prompts at T=0 / 128 tokens; texts must equal the reference's; ms/round (median of
#            decode seconds / MTP rounds) is reported against it (PASS within +-1 %, else WARN)
#   dflash2  a short DFlash2 K=7 --lm-head-draft run on the _df2nv artifact (10 prompts), same checks
#
# The gate does not reserve the GPUs; run it on idle GPUs or wrap it in your own GPU scheduler.
# Artifact paths default to $GATE_ARTIFACTS_DIR (default $HOME/ninfer-artifacts); override any of
# them with the GATE_* variables below.
# GATE_EXTRA_FLAGS (e.g. --embedding-host) is appended to the runner of golden, ppl, greedy and
# dflash2, so an option that must not change any result is checked against the same reference.
set -uo pipefail

here=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
src=${GATE_SRC:-$(cd "$here/../.." && pwd)}   # source tree: golden record.sh, perplexity corpus

ARTIFACTS_DIR=${GATE_ARTIFACTS_DIR:-$HOME/ninfer-artifacts}
ARTIFACT=${GATE_ARTIFACT:-$ARTIFACTS_DIR/qwen3_8_27b_quasar_nvfp4_mtpnv.ninfer}
# The production DFlash2 artifact (drafter MLP gate/up NVFP4, v0.4.5); the older _df2 (Q8) stays for comparisons.
DF2_ARTIFACT=${GATE_DF2_ARTIFACT:-$ARTIFACTS_DIR/qwen3_8_27b_quasar_nvfp4_df2nv.ninfer}
TEST_ARTIFACT=${GATE_TEST_ARTIFACT:-$DF2_ARTIFACT}
TEST_SCRATCH=${GATE_TEST_SCRATCH:-$ARTIFACTS_DIR/tmp}
GOLDEN_MODEL=${GATE_GOLDEN_MODEL:-$ARTIFACTS_DIR/tmp/golden-sync/synthetic/model.ninfer}
PROMPTS=${GATE_PROMPTS:-}           # --record only: JSONL of {"id","text"}; copied into the reference
DEVICES=${GATE_DEVICES:-0,1}
PORT=${GATE_PORT:-8090}
CORPUS=${GATE_CORPUS:-$src/eval/corpora/perplexity-1m/manifest.json}
SERVE_FLAGS=${GATE_SERVE_FLAGS:---tp 2 --devices $DEVICES --kv-dtype int8 --max-context 196608 --kv-capacity 196608 --device-state-slots 4 --max-concurrency 1 --spec mtp --draft-tokens 3 --lm-head-draft --vision --vision-device 0 --max-vision-tokens 4096}
DF2_FLAGS=${GATE_DF2_FLAGS:---tp 2 --devices $DEVICES --kv-dtype int8 --max-context 32768 --kv-capacity 32768 --max-concurrency 1 --spec dflash2 --draft-tokens 7 --lm-head-draft}
DF2_PROMPTS=${GATE_DF2_PROMPTS:-10}
EXTRA_FLAGS=${GATE_EXTRA_FLAGS:-}
PPL_PAIRED=${GATE_PPL_PAIRED:-0}    # 1: a perplexity table that differs is judged by ppl_paired.py, not FAIL
PPL_ALPHA=${GATE_PPL_ALPHA:-0.05}; PPL_MAX_DELTA=${GATE_PPL_MAX_DELTA:-0.5}
# Tests that cannot run on one 16 GB board (tp1 27B artifacts) or need a different artifact.
CTEST_EXCLUDE=${GATE_CTEST_EXCLUDE:-ninfer_qwen3_5_(prefix|score|moe|dflash|dflash2|dflash_prefill)_real_test|ninfer_qwen3_5_vision_workspace_test}

record=0; stages="shards,ctest,golden,ppl,greedy,dflash2"; attention=0; out=""
while [ $# -gt 0 ]; do
    case $1 in
        --record) record=1; shift ;;
        --stages) stages=$2; shift 2 ;;
        --attention) attention=1; shift ;;
        --out) out=$2; shift 2 ;;
        -h|--help) sed -n '2,27p' "$0"; exit 0 ;;
        -*) echo "unknown option $1" >&2; exit 2 ;;
        *) break ;;
    esac
done
[ $# -eq 2 ] || { echo "usage: $0 [--record] [--stages LIST] [--attention] [--out DIR] <build-dir> <reference-dir>" >&2; exit 2; }
build=$(cd "$1" && pwd); ref=$2
mkdir -p "$ref"; ref=$(cd "$ref" && pwd)
[ -n "$out" ] || out=$build/gate/$(date +%Y%m%d-%H%M%S)
[ $record = 1 ] && out=$ref
mkdir -p "$out"; out=$(cd "$out" && pwd)
summary=$out/summary.txt; : > "$summary"
failed=0

say() { echo "[$(date '+%F %T')] $*" | tee -a "$summary"; }
verdict() { # stage status detail
    say "$(printf '%-8s %-5s %s' "$1" "$2" "$3")"
    [ "$2" = FAIL ] && failed=1
    return 0
}
has() { [[ ",$stages," == *",$1,"* ]]; }
gpu_busy() { nvidia-smi --query-compute-apps=pid --format=csv,noheader 2>/dev/null | grep -q .; }

say "gate $([ $record = 1 ] && echo RECORD || echo CHECK) | build $build | ref $ref | out $out | src $(git -C "$src" rev-parse --short=8 HEAD 2>/dev/null) dirty=$(git -C "$src" status --porcelain 2>/dev/null | wc -l) | stages $stages"
say "clocks $(nvidia-smi --query-gpu=clocks.sm --format=csv,noheader 2>/dev/null | tr '\n' ' ')"
[ -n "$EXTRA_FLAGS" ] && say "extra flags: $EXTRA_FLAGS"
[ "$PPL_PAIRED" = 1 ] && say "ppl: paired mode (alpha $PPL_ALPHA, max |change| $PPL_MAX_DELTA %)"

# ---------------------------------------------------------------- build
if has build; then
    t0=$(date +%s)
    if cmake --build "$build" -j"${GATE_JOBS:-12}" > "$out/build.log" 2>&1; then
        verdict build PASS "$(( $(date +%s) - t0 )) s, $(grep -c warning "$out/build.log") warnings"
    else
        verdict build FAIL "see $out/build.log"; say "RESULT FAIL"; exit 1
    fi
fi

# ---------------------------------------------------------------- shards
if has shards; then
    if python3 "$src/tools/tp2/check_shard_sources.py" "$src" > "$out/shards.log" 2>&1; then
        verdict shards PASS "$(tail -1 "$out/shards.log")"
    else verdict shards FAIL "see $out/shards.log"; fi
fi

# ---------------------------------------------------------------- ctest
if has ctest; then
    t0=$(date +%s)
    exclude=$CTEST_EXCLUDE
    [ $attention = 1 ] || exclude="$exclude|ninfer_softmax_attention_test"
    NINFER_TEST_ARTIFACT=$TEST_ARTIFACT NINFER_TEST_SCRATCH_DIR=$TEST_SCRATCH \
        ctest --test-dir "$build" -j1 --timeout 900 --output-on-failure -E "$exclude" > "$out/ctest.log" 2>&1
    rc=$?
    line=$(grep -E 'tests passed|tests failed' "$out/ctest.log" | tail -1)
    tp2=$(grep -cE 'tp2.*Passed' "$out/ctest.log")
    if [ $rc = 0 ]; then verdict ctest PASS "$line; tp2 tests passed: $tp2; $(( $(date +%s) - t0 )) s"
    else verdict ctest FAIL "$line; $(grep -E '\*\*\*|Failed' "$out/ctest.log" | grep -oE 'ninfer_[a-z0-9_]+' | sort -u | tr '\n' ' ')"; fi
fi

# ---------------------------------------------------------------- golden
if has golden; then
    g=$out/golden; rm -rf "$g"
    if CUDA_VISIBLE_DEVICES=${DEVICES%%,*} GOLDEN_FLAGS=$EXTRA_FLAGS "$src/tools/golden/record.sh" "$build/apps/ninfer-tp1-golden" "$GOLDEN_MODEL" "$g" > "$out/golden.log" 2>&1; then
        if [ $record = 1 ]; then verdict golden REC "$(tr '\n' ' ' < "$out/golden.log")"
        elif diff -r "$ref/golden" "$g" > "$out/golden.diff" 2>&1; then
            verdict golden PASS "3/3 identical ($(cat "$g"/*.ids | md5sum | cut -c1-8))"
        else verdict golden FAIL "ids differ, see $out/golden.diff"; fi
    else verdict golden FAIL "runner failed, see $out/golden.log"; fi
fi

# ---------------------------------------------------------------- ppl
if has ppl; then
    for p in "64k 65536 32768" "4k 4096 2048"; do
        read -r tag ctx stride <<< "$p"
        d=$out/ppl-$tag; rm -rf "$d"; mkdir -p "$d"
        t0=$(date +%s)
        # shellcheck disable=SC2086
        "$build/apps/ninfer-perplexity" "$ARTIFACT" --corpus "$CORPUS" --quick --context "$ctx" --stride "$stride" \
            --tp 2 --devices "$DEVICES" --kv-dtype int8 $EXTRA_FLAGS --output "$d" > "$d.log" 2>&1
        rc=$?
        # The per-source table and the overall line, without the rate and paths.
        awk 'f{print} /^domain/{f=1} /^overall/{exit}' "$d.log" > "$d.table"
        ov=$(grep -E '^overall' "$d.log" | awk '{print $NF}')
        if [ $rc != 0 ] || [ -z "$ov" ]; then verdict "ppl-$tag" FAIL "rc=$rc, see $d.log"
        elif [ $record = 1 ]; then
            # The per-window report is what the paired mode compares against.
            cp "$d/report.json" "$ref/ppl-$tag.report.json" 2>/dev/null
            verdict "ppl-$tag" REC "$ov ($(( $(date +%s) - t0 )) s)"
        elif diff "$ref/ppl-$tag.table" "$d.table" > /dev/null; then
            verdict "ppl-$tag" PASS "$ov, $(wc -l < "$d.table") lines identical ($(( $(date +%s) - t0 )) s)"
        elif [ "$PPL_PAIRED" = 1 ]; then
            if [ ! -f "$ref/ppl-$tag.report.json" ] || [ ! -f "$d/report.json" ]; then
                verdict "ppl-$tag" FAIL "paired mode needs $ref/ppl-$tag.report.json and $d/report.json"
            else
                python3 "$here/ppl_paired.py" --alpha "$PPL_ALPHA" --max-delta-pct "$PPL_MAX_DELTA" \
                    "$ref/ppl-$tag.report.json" "$d/report.json" > "$d.paired" 2>&1
                res=$(head -1 "$d.paired")
                case ${res%% *} in
                    PASS|WARN) verdict "ppl-$tag" "${res%% *}" "${res#* } ($(( $(date +%s) - t0 )) s)" ;;
                    *) verdict "ppl-$tag" FAIL "paired comparison: ${res:-no output}, see $d.paired" ;;
                esac
            fi
        else verdict "ppl-$tag" FAIL "$ov vs $(grep -E '^overall' "$ref/ppl-$tag.table" | awk '{print $NF}')"; fi
    done
fi

# ---------------------------------------------------------------- serve stages
serve_pid=""
serve_down() {
    [ -n "$serve_pid" ] || return 0
    kill "$serve_pid" 2>/dev/null
    for _ in $(seq 1 60); do kill -0 "$serve_pid" 2>/dev/null || break; sleep 0.5; done
    kill -9 "$serve_pid" 2>/dev/null; wait "$serve_pid" 2>/dev/null; serve_pid=""; sleep 2
}
trap serve_down EXIT
serve_up() { # tag artifact flags...
    local tag=$1 art=$2; shift 2
    if gpu_busy; then say "$tag: GPU not empty: $(nvidia-smi --query-compute-apps=pid,name --format=csv,noheader | tr '\n' ' ')"; return 1; fi
    rm -f "$out/req-$tag.jsonl"
    "$build/apps/ninfer-serve" "$art" --host 127.0.0.1 --port "$PORT" --model-id gate "$@" \
        --request-log-jsonl "$out/req-$tag.jsonl" > "$out/serve-$tag.log" 2>&1 &
    serve_pid=$!
    for _ in $(seq 1 600); do
        [ "$(curl -s -o /dev/null -w '%{http_code}' -m 3 "127.0.0.1:$PORT/health")" = 200 ] && return 0
        kill -0 "$serve_pid" 2>/dev/null || { say "$tag: server died, see $out/serve-$tag.log"; serve_pid=""; return 1; }
        sleep 0.5
    done
    say "$tag: no health"; serve_down; return 1
}
serve_stage() { # tag artifact limit flags...
    local tag=$1 art=$2 limit=$3; shift 3
    local t0; t0=$(date +%s)
    if ! serve_up "$tag" "$art" "$@"; then verdict "$tag" FAIL "server did not start"; return; fi
    python3 "$here/gate_client.py" run "127.0.0.1:$PORT" "$ref/prompts.jsonl" "$limit" "$out/$tag.jsonl" > "$out/$tag.client.log" 2>&1
    local rc=$?
    serve_down
    if [ $rc != 0 ]; then verdict "$tag" FAIL "client rc=$rc, see $out/$tag.client.log"; return; fi
    local res
    if [ $record = 1 ]; then
        res=$(python3 "$here/gate_client.py" summarize "$out/$tag.jsonl" "$out/req-$tag.jsonl")
        verdict "$tag" REC "$res ($(( $(date +%s) - t0 )) s)"
    else
        res=$(python3 "$here/gate_client.py" compare "$ref/$tag.jsonl" "$ref/req-$tag.jsonl" "$out/$tag.jsonl" "$out/req-$tag.jsonl")
        local st=${res%% *}
        verdict "$tag" "$st" "${res#* } ($(( $(date +%s) - t0 )) s)"
    fi
}
if [ $record = 1 ] && { has greedy || has dflash2; }; then
    [ -n "$PROMPTS" ] || { say "--record needs GATE_PROMPTS"; exit 2; }
    cp "$PROMPTS" "$ref/prompts.jsonl"
fi
# shellcheck disable=SC2086
has greedy && serve_stage greedy "$ARTIFACT" 0 $SERVE_FLAGS $EXTRA_FLAGS
# shellcheck disable=SC2086
has dflash2 && serve_stage dflash2 "$DF2_ARTIFACT" "$DF2_PROMPTS" $DF2_FLAGS $EXTRA_FLAGS

if [ $record = 1 ]; then say "RESULT RECORDED"; exit $failed; fi
[ $failed = 0 ] && say "RESULT PASS" || say "RESULT FAIL"
exit $failed

tools/tp2/README.md

tools/tp2

• gate.sh - the behavior-preservation gate of the two-GPU layer: one command, PASS/FAIL per
  stage against a recorded reference. Mandatory for every upstream merge and every change to the
  tp2 layer (Merging upstream).
• gate_client.py - its serve-stage client (requests, request-log statistics, comparison).
• check_shard_sources.py - the static guard of the shards compiled from upstream's sources.
• ppl_paired.py - the paired per-window comparison of two ninfer-perplexity reports, which the
  ppl stage runs with GATE_PPL_PAIRED=1.
• surface.sh - the fork's footprint inside upstream's files: modified files, lines, hunks, the
  files with the most hunks (tools/tp2/surface.sh [<upstream-ref> [<ref>]]).
• mailbox_probe.cu - the standalone pinned-host mailbox check (Tools).

gate.sh

tools/tp2/gate.sh [--record] [--stages LIST] [--attention] [--out DIR] <build-dir> <reference-dir>

Stage	Runs	Passes when
build (not in the default list)	cmake --build <build-dir>	it builds
shards	tools/tp2/check_shard_sources.py on the source tree, no GPU	no upstream definition outside an anonymous namespace in a source a shard re-includes, no renamed identifier in a header the shard's prelude does not include first
ctest	every CTest except the tp1 27B tests that do not fit one 16 GB board and, without --attention, ninfer_softmax_attention_test (~12 min); NINFER_TEST_ARTIFACT is the _df2nv artifact	all pass (skips allowed)
golden	tools/golden/record.sh on the synthetic tp1 model	ids equal the reference's
ppl	ninfer-perplexity --tp 2 --quick on the QUASAR artifact, INT8 KV, 65536/32768 and 4096/2048	the per-source table equals the reference's to every printed digit; with GATE_PPL_PAIRED=1 a table that differs is judged window by window (below)
greedy	ninfer-serve with the production flags (MTP3, --lm-head-draft, C=1) on the _mtpnv artifact (production since v0.4.7: the MTP layer's MLP and attention output are NVFP4, configure_mtp_nvfp4), the reference's 60 prompts, T=0, 128 tokens	texts identical; ms/round (median of decode seconds per MTP round) PASS within ±1 % of the reference, WARN beyond
dflash2	ninfer-serve DFlash2 K=7 --lm-head-draft on the _df2nv artifact (production since v0.4.5: the drafter MLP gate/up is NVFP4), first 10 prompts	as greedy

--record writes the reference directory from the given build (it needs GATE_PROMPTS, a JSONL
of {"id","text"}, copied into the reference). Every path has a GATE_* override
(GATE_ARTIFACT, GATE_DF2_ARTIFACT, GATE_TEST_ARTIFACT, GATE_GOLDEN_MODEL, GATE_CORPUS,
GATE_DEVICES, GATE_PORT, GATE_SERVE_FLAGS, GATE_DF2_FLAGS, GATE_SRC, GATE_ARTIFACTS_DIR);
artifacts default to $GATE_ARTIFACTS_DIR ($HOME/ninfer-artifacts). The script does not reserve the
GPUs: run it on idle GPUs or wrap it in your own scheduler. Output and summary.txt go to
<build-dir>/gate/<timestamp>/ or --out. About 13 minutes on two RTX 5070 Ti.

A WARN on ms/round is not a verdict: rerun the two builds alternated (A B B A) before deciding.

Paired perplexity for commits that change bits

The default ppl check wants every printed digit of the reference's table, which is right for a
change that must not move a bit. For a commit that changes bits on purpose (a rounding fix, a
schedule with another K order, a different quantization) that check can only fail, and a fixed
threshold on the overall perplexity does not help: one standard error of the overall perplexity
is about 0.06 % at 4096/2048 and 0.17 % at 65536/32768 on the full corpus (more on --quick), so
any threshold below that passes or fails by chance. GATE_PPL_PAIRED=1 replaces it with a paired
test. Each window of the report (--output, report.json) is a fresh prefill scoring its own
disjoint tokens, so the per-window differences of total NLL are samples of the perturbation;
ppl_paired.py takes the mean difference per scored token, its standard error from the spread
between windows, the two-sided p of the paired t (n - 1 degrees of freedom), the 95 % interval of
the perplexity change and an exact sign test.

Outcome	Verdict
table identical to every digit	PASS (as without the option)
not worse at GATE_PPL_ALPHA (better, or worse with p > 0.05) and |change| < GATE_PPL_MAX_DELTA (0.5 %)	PASS, with the change, interval, p and signs
significantly worse, or |change| ≥ 0.5 % either way	WARN
the reference has no report, or the reports score different windows	FAIL

--record stores each report next to its table (ppl-64k.report.json, ppl-4k.report.json). The
quick 64k plan has only 4 windows: its test has little power (it flags only large changes), the
4k plan (124 windows) is the informative one; for a decision, run the full corpus (16 / 496 windows,
ninfer-perplexity without --quick) on both builds and compare the reports with ppl_paired.py.
The paired mode does not judge task quality: a commit that changes bits also takes a paired task
check outside the gate, GSM8K with lm-eval on both builds and an exact McNemar test on the
discordant problems (the protocol of the residual-rounding fix: 1 319 or a fixed first-N subset,
5-shot, thinking on, T=0, max_gen_toks 8192; PASS at p > 0.05).

The recorded reference

reference/ is the reference the gate compares against on the development pair (two RTX 5070 Ti
without P2P, eco clocks at 2085 MHz, nvcc 13.2, sm_120a, the default GATE_* artifacts):
tools/tp2/gate.sh <build> tools/tp2/reference. It holds only what the stages compare (the
prompts, the texts and request logs of greedy and dflash2, the perplexity tables and their
per-window reports, the golden ids) and the recording summary (summary.txt). It was recorded from fix/nvfp4-residual-rounding
(c3383145: the A4 fused residual update rounds as a product then a sum in every tile; v0.4.5 contracted
it into one FMA only in launches whose width filled every token tile) with the _df2nv DFlash2
artifact. The golden ids and both perplexity tables equal those of v0.3.0-v0.4.5 to every digit,
greedy differs from the previous reference in one text (gsm35), and dflash2 on _df2nv gives the
same ten texts as _df2 did (acceptance 62.66 %, 18.200 ms/round). The two perplexity reports
(ppl-*.report.json) are that recording's own, added with the paired mode. The previous references (main
169514ea for golden, perplexity and greedy; fdf6c33a for dflash2 on _df2) are in this directory's
history. Another board has other texts: record its own reference (--record) rather than comparing
against this one.