openTPU

openTPU

An open-source AI accelerator, developed by AI.

openTPU brings the lessons of auto-arch-tournament
to AI accelerators. It asks two questions: how far can AI agents go at hardware design, and can
they build the chip that runs their own inference?

otpu-chat running LFM2.5-230M on the FPGA card (left), with otpu-smi showing the card's
utilization and DRAM bandwidth (right).

A place to learn

openTPU is also a learning project. The whole accelerator lives in one small monorepo that you
can read end to end: the hardware design (SystemVerilog), the instruction set, a bit-exact
simulator, a kernel language and its compiler, and the host software that drives a real PCIe
card. If you want to understand how an AI accelerator works, from a matmul in Python down to
the wires, this is a good place to start.

Results

The design runs ten modern models with their real weights on an Inspur YPCB-00338 card
(Xilinx Kintex-7 xc7k480t, two DDR3 channels), and the card produces the same tokens as the
simulator, bit for bit.

Model	Weights	Decode, device	Decode, wall	Prefill, device	DRAM while decoding
LFM2.5-230M	int8	59.0 tok/s	52.3 tok/s	295.6 tok/s	14.5 GB/s (85% of peak)
LFM2.5-230M	4-bit, int8 head	85.8 tok/s	82.1 tok/s	335.4 tok/s	14.1 GB/s (82%)
Qwen3-0.6B	int8	21.6 tok/s	21.3 tok/s	92.1 tok/s	14.4 GB/s (84%)
Qwen3-0.6B	4-bit, int8 head	31.3 tok/s	30.7 tok/s	103.4 tok/s	13.9 GB/s (82%)
Qwen3.5-0.8B	int8	17.6 tok/s	16.3 tok/s	61.4 tok/s	14.5 GB/s (85%)
Qwen3.5-0.8B	4-bit, int8 head	24.5 tok/s	23.3 tok/s	66.7 tok/s	14.1 GB/s (83%)
Gemma 4 E2B	4-bit, int8 head	10.57 tok/s	10.53 tok/s	32.1 tok/s	15.6 GB/s (92%)
Gemma 4 E2B	4-bit, 4-bit head	12.14 tok/s	12.09 tok/s	29.9 tok/s	15.5 GB/s (91%)
LFM2-2.6B	int8	6.05 tok/s	6.03 tok/s	21.4 tok/s	16.1 GB/s (94%)
LFM2-2.6B	4-bit, int8 head	10.96 tok/s	10.93 tok/s	20.6 tok/s	15.8 GB/s (93%)
SmolLM3-3B	int8	5.00 tok/s	4.99 tok/s	21.1 tok/s	16.0 GB/s (94%)
SmolLM3-3B	4-bit, int8 head	8.74 tok/s	8.72 tok/s	22.8 tok/s	15.7 GB/s (92%)
Phi-4-mini (3.8B)	int8	3.99 tok/s	3.98 tok/s	13.8 tok/s	16.0 GB/s (94%)
Phi-4-mini (3.8B)	4-bit, int8 head	6.56 tok/s	6.55 tok/s	15.0 tok/s	15.8 GB/s (92%)
Qwen3.5-2B	int8	8.02 tok/s	8.00 tok/s	38.2 tok/s	16.0 GB/s (94%)
Qwen3.5-2B	4-bit, int8 head	12.09 tok/s	12.03 tok/s	41.7 tok/s	15.8 GB/s (92%)
Qwen3.5-4B	4-bit, int8 head	5.88 tok/s	5.87 tok/s	12.9 tok/s	15.7 GB/s (92%)
Gemma 4 E4B	int8, 4-bit head and down 0-23	3.78 tok/s	3.75 tok/s	14.8 tok/s	16.0 GB/s (94%)

Measured on the card: the first three models on 2026-09-29 with the production image
deploy_champ_e698dcd7. LFM2-2.6B, SmolLM3-3B and Phi-4-mini on 2026-09-30, and Qwen3.5-2B and
4B and Gemma 4 on 2026-10-01, with build B, deploy_fused133c_79c5707a, production since then.
Build B decodes LFM2-2.6B, SmolLM3 and Phi-4-mini 8-9% faster than e698dcd7 (Gemma 4 E2B 10%),
at 91-94% of the DRAM peak instead of 82-87%. Qwen3.5-4B's int8 image is over 4 GiB.

• The image: main e698dcd at 133.33 MHz, one bitstream for all models. It has LiteDRAM
  controllers calibrated by a small CPU inside the memory core, a four-column systolic matrix
  unit and the stream engine (docs/stream.md). DDR3-1066, with a 17.1 GB/s
  peak.
• The host: the card sits in opentpu (Intel Core i7-4790).
• Method, tools/qual/perf.py: decode is 64 greedy tokens after a 512-token prompt, with the
  host's argmax in the loop (not streamed). "Device" counts only the cycles the accelerator runs;
  "wall" adds the host. Prefill is the 512-token prompt, on the device.
• DRAM traffic comes from the card's own counters while it runs.
• Gemma 4 E2B keeps its per-layer embedding tables on the card (3.5-3.6 GiB images;
  docs/gemma4.md); in int8 it does not fit. It matches Hugging Face's greedy
  tokens on three prompts with either head. E4B's table (2.95 GB) stays on the host, which
  copies one 11 KB row into the card per token; its image is 3.96 GiB, int8 with the head and
  the first 24 layers' down projections in 4-bit (docs/gemma4_e4b.md).
  In the card's own decode loop (the card picking every token) Gemma 4 decodes faster: E2B
  11.01 / 12.73 tok/s (int8 / 4-bit head), E4B 3.83 tok/s, on the device.
• Every configuration matches the simulator token for token, per-position and with the
  resident decode program. More detail in docs/board.md.

With the logits streamed back while the card runs (tools/decode_profile.py, 96 tokens), 4-bit
decode is faster, in device / wall tok/s:

• LFM2: 89.5 / 84.5;
• Qwen3: 33.7 / 33.3;
• Qwen3.5: 24.6 / 24.2;
• LFM2-2.6B: 11.07 / 11.02 (build B);
• SmolLM3-3B: 8.92 / 8.89 (build B);
• Phi-4-mini: 6.69 / 6.67 (build B).

The previous production image, se-cand3, was built with the Xilinx MIG, a two-column matrix
unit and a 120.755 MHz clock. Measured the same way, the new image:

• decode: within 2.3% of se-cand3's in every configuration. Decode is bound by DRAM, and
  LiteDRAM reads at 82-85% of the DDR3 peak, as the MIG did.
• prefill: 1.3x (Qwen3.5) to 2.0x (LFM2 4-bit) faster.
• calibration: when the image starts, the core's CPU calibrates both DDR3 channels in 12 s,
  with no host involvement.

The earlier images and their numbers are in docs/board.md, section 5.

Mixture-of-experts models bigger than the card's 4 GiB run with their experts streamed from host
storage (docs/offload.md, section 10). The card routes each token and computes
every expert, and it keeps the experts in per-layer slots in its DRAM. The host only copies
missing experts from a pool file into those slots, at the link's rate (section 10.1). Measured
on 2026-10-01 with build B (79c5707a), the card's own decode loop picking every token, 4-bit
experts, int8 head:

• LFM2.5-8B-A1B (8.5B parameters, 1.7B active): 10.6 tok/s over 160 tokens. 98.5% of expert
  uses hit the slots, and 5.2 MB streamed per token.
• Qwen3.5-35B-A3B (34.7B parameters, 3.0B active): 3.95 tok/s, with Hugging Face's 16
  greedy tokens. 62% of expert uses hit, and 153 MB streamed per token at 1.41 GB/s over PCIe
  (section 10.3).
• Both match the simulator bit for bit.

4-bit weights (docs/quant.md) use FP4 values with two-level block scales, 4.25
bits per weight, and keep the LM head in int8 for accuracy. They cut the bytes per token by about
a third and raise decode speed by 40% (Qwen3.5) to 45% (Qwen3, LFM2), at a measurable cost in
perplexity that docs/quant.md reports per model.

The host is nearly out of the way. For LFM2 and Qwen3 the card runs one decode program compiled
once, which reads the position from a register and looks up its own embedding and RoPE rows, and
the logits stream back while the card is still running: the host adds 0.17 to 0.30 ms per
token on omarchy (0.45 to 1.3 ms on opentpu).
Qwen3.5 runs the same way for decode; its prefill still compiles each chunk's program on the host,
ahead of the card.

How it works

  Kernels in ol              mlp, attention, full model layers
        |  @ol.jit
  Language + compiler        layouts, affine loop addressing, fusion
        |
  ISA                        8 x 32-bit words per instruction
        |
  ISA simulator  <======>  RTL          same bits, checked by the tests
  (Python)                 (SystemVerilog)
                            |  Vivado bitstream
                           FPGA card    Kintex-7 xc7k480t
                            |  PCIe
                           Host         otpu-chat, otpu-smi, otpu-lens

The machine is deliberately simple. A sequencer issues one instruction per cycle to a few
units: DMA moves data, the matrix unit multiplies int8 weights streamed from DRAM, the vector
unit does fp32 math, and a quantizer turns results back into int8. There is no cache and no
hidden scheduling: every data movement is an instruction, so a trace shows exactly where the
cycles go. docs/isa.md describes the whole instruction set.

A kernel looks like this:

from opentpu import language as ol

@ol.jit
def mlp(h, gamma, w_gate, w_up, w_down, out, eps):   # simplified; see kernels/mlp.py
    x = ol.load(h)
    xs = ol.quantize(rmsnorm(x, ol.load(gamma), eps))
    g = ol.dot(xs, w_gate)
    u = ol.dot(xs, w_up)
    a = ol.all_gather(silu(g) * u)
    y = ol.all_gather(ol.dot(a, w_down))
    if ol.program_id() == 0:
        ol.store(out, x + y)

Because every data movement is an instruction, a trace of a run explains its speed. Lens, the
profiler, records a run from the RTL, the simulator or the card and opens it in the browser,
with a roofline, a timeline and per-instruction tables (docs/lens.md).

Lens replaying part of a Qwen3 decode step. Colours show what each unit is doing in each
cycle: busy, waiting on DRAM, or waiting on another instruction.

Try it

Everything except the card runs on a laptop.

pip install -e .
pip install pytest torch transformers
python3 -m pytest -q          # RTL tests also need Verilator 5

hf download LiquidAI/LFM2.5-230M --local-dir models/LFM2.5-230M
otpu-chat --model lfm2 --backend isa    # chat on the simulator

With a card, build the bitstream (make bit in boards/ypcb-00338), load
it over JTAG, then run sudo otpu-setup and otpu-chat --backend board.
docs/board.md walks through the bring-up.

Command	What it does
otpu-chat	chat with Qwen3-0.6B, LFM2.5-230M (--model lfm2), Qwen3.5-0.8B (--model qwen35), LFM2-2.6B (lfm2-2.6b), SmolLM3-3B (smollm3), Phi-4-mini (phi4-mini) or Qwen3.5-2B / 4B (qwen35-2b, qwen35-4b)
otpu-smi	temperature, power, DRAM bandwidth and per-unit utilization
otpu-lens	record a run and open it in the profiler
otpu-selftest, otpu-diag	check that the card works

Where to start reading

1. docs/isa.md: the instruction set. Everything else is built on it.
2. opentpu/kernels and docs/compiler.md: how a kernel
   becomes instructions.
3. opentpu/isasim.py: the simulator, which is the spec.
4. rtl/: the hardware, starting from rtl/top/otpu_top.sv.
5. docs/lfm2.md, docs/qwen35.md, docs/llama.md,
   docs/benchmarks.md: whole models and where their cycles go.
6. docs/board.md: the physical card, from clocks to PCIe.

What's next

• The last few percent of DRAM. Decode is bound by DRAM efficiency: it reads 82 to 85% of the
  DDR3-1066 peak. Work on the LiteDRAM path's efficiency is under way.
• Timing margin and area. The design closes 133.33 MHz, the clock at which the 128-byte port
  matches the two DDR3 channels, but only just (WNS +0.032 ns). A tournament of Vivado runs keeps
  working on its margin and area. Decode is bound by DRAM, so a faster clock mostly helps prefill.
• Faster prefill. The four-column systolic matrix unit is in the production image; prefill is
  still limited by the matrix unit's multiply rate.

tools/qual/perf.py

"""Card qualification: prefill tok/s on a long prompt, then decode with the card's free-running
counters (device cycles, DRAM traffic and efficiency, MXU starvation) and wall time.

    python3 tools/qual/perf.py MODEL WFORMAT [HEAD_FORMAT] [--prompt 512] [--tokens 64]

Run from a host tree (the repo root). One engine per call, with resident decode where the model
has it (the check that resident decode computes the ISA simulator's tokens is refs.py card
--resident, after the warm soak).
"""
from __future__ import annotations

import argparse
import sys
import time
from pathlib import Path

import numpy as np

ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(ROOT))

from transformers import AutoTokenizer  # noqa: E402

from opentpu.host import chat as C  # noqa: E402
from opentpu.llm import load_spec, model_dir  # noqa: E402
from opentpu.llm import qwen3 as Q  # noqa: E402


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("model")
    ap.add_argument("wformat")
    ap.add_argument("head_format", nargs="?")
    ap.add_argument("--prompt", type=int, default=512, help="prefill tokens")
    ap.add_argument("--tokens", type=int, default=64, help="decode tokens")
    a = ap.parse_args()
    m, wf, hf = a.model, a.wformat, (a.head_format if a.head_format not in (None, "-") else None)
    path = model_dir(m)
    spec, tok = load_spec(path), AutoTokenizer.from_pretrained(path)
    backend, cfg = C.make_backend("board", spec, 2048, "/dev/xdma0", path.name)
    eng = Q.Engine(spec, Q.load_weights(path), cap=2048, cfg=cfg, backend=backend,
                   wformat=wf, head_format=hf, resident=True)
    board, info = eng.backend.board, eng.backend.info
    khz = info["core_khz"]
    text = (ROOT / "tools/data/austen_pp_ch1.txt").read_text()
    ids = tok(text)["input_ids"][:a.prompt]
    t0, n0 = time.perf_counter(), len(eng.stats)
    logits = eng.prefill(ids)
    wall = time.perf_counter() - t0
    pcyc = sum(s["cycles"] for s in eng.stats[n0:])
    print(f"{path.name} ({wf}{', head ' + hf if hf else ''}): prefill {len(ids)} tokens in "
          f"{len(eng.stats) - n0} runs: device {len(ids) * khz * 1e3 / pcyc:.1f} tok/s, "
          f"wall {len(ids) / wall:.1f} tok/s (compile included); resident={eng.resident}")
    t = int(np.argmax(logits))
    n, s0 = a.tokens, board.snapshot()
    n0 = len(eng.stats)
    t0 = time.perf_counter()
    for _ in range(n):
        t = int(np.argmax(eng.step(t)))
    eng._drain()
    wall = time.perf_counter() - t0
    s1 = board.snapshot()
    dev = sum(s["cycles"] for s in eng.stats[n0:])
    d = {k: s1[k] - s0[k] for k in ("DRAM_RD", "DRAM_WR", "RUNNING", "MXU_STARVE", "DRAM_WAIT")
         if k in s0}
    run_s = d["RUNNING"] / (khz * 1e3)
    peak = 2 * 8 * info["ddr_mts"] * 1e6 / 1e9 if info.get("ddr_mts") else 17.06
    gbs = (d["DRAM_RD"] + d["DRAM_WR"]) * 64 / run_s / 1e9
    print(f"  decode {n} tokens: {dev / n / 1e6:.3f} Mcycles/token, device "
          f"{n * khz * 1e3 / dev:.2f} tok/s, wall {n / wall:.2f} tok/s (argmax loop, not "
          f"streamed); DRAM read {d['DRAM_RD'] * 64 / n / 1e6:.1f} MB/token, write "
          f"{d['DRAM_WR'] * 64 / n / 1e6:.2f} MB/token; while running {gbs:.2f} GB/s "
          f"(read {d['DRAM_RD'] * 64 / run_s / 1e9:.2f}) = {100 * gbs / peak:.0f}% of "
          f"{peak:.1f} GB/s"
          + (f"; MXU_STARVE {100 * d['MXU_STARVE'] / d['RUNNING']:.0f}%" if "MXU_STARVE" in d else "")
          + (f"; DRAM_WAIT {100 * d['DRAM_WAIT'] / d['RUNNING']:.0f}%" if "DRAM_WAIT" in d else ""))
    eng.backend.close()


if __name__ == "__main__":
    main()

tools/decode_profile.py

"""Where the host time of one decode token goes: a timeline of otpu-chat's decode loop.

    python3 tools/decode_profile.py --model lfm2 [--backend board | fake] [--tokens 64]
                                    [--prompt "..."] [--greedy] [--no-stream] [--json out.json]
                                    [--wformat int8|fp4|int4] [--head-format int8|fp4|int4]
                                    [--card-loop]

Runs one Chat turn (plain mode, the reply printed to /dev/null) on the card (--backend
board) or on FakeTransport (--backend fake: a card that computes nothing and halts after
--fake-ms; the host path only, without streamed logits).

The number that matters is the host's critical path per token: from the moment the host sees
HALTED to the moment it writes RUN for the next token (the card idles in between). The rest of
the host's work overlaps a run. Every transport operation (DMA write / read, register read /
write, the HALTED poll with its sleeps) is timed and filed under the host step it serves, as
"critical" (between HALTED and RUN) or "overlapped" (while the card runs):

  io-write        x / cos / sin of the next token (BoardBackend.write)
  prog-upload     the program's DMA to the program area (inside Board.load_program)
  imem-load       LOAD .. not LOADING (Board.load_program's registers and poll)
  start           CLEAR, RUN (and the trace registers)
  counters        the HALTED poll and the counter registers after it (Board.wait); its critical
                  part starts when the run ends: the poll's wake-up and the register reads
  logits-stream   the logits pieces read while the card runs (streamed logits); logits-tail:
                  what is read after HALTED; logits-mark: waiting for their sentinel marking
                  (after the run, on the DMA worker, whose calls count as "(worker)" operations)
  logits-read     the logits read after HALTED without streaming (--no-stream)
  sample, detok, ui, status   host computation (no transport)
  compile-wait    the step waiting for the precompiled program (Engine._program)
  other           the critical window's time not in any item above

Prints the mean per decode token (the first generated token and the prefill are excluded),
the transport operations per token (count, bytes, time) and wall vs device tokens/s.

--card-loop: the decode loop on the card (docs/autodecode.md, Chat.on_card; a bitstream with
CAPS bit30): one run for the reply, so there is no per-token host path to split; prints the
tokens, the runs and wall vs device tokens/s after the host's pick of the first token (the
run's start included: the bucket's compile, the program's upload, the sampler's inputs), and
the rate from the first token the card picked to its last, with the time before that first
token and the compile's part of it.
"""
from __future__ import annotations

import argparse
import io
import json
import sys
import threading
import time
from collections import defaultdict
from pathlib import Path

ROOT = Path(__file__).resolve().parent.parent
sys.path.insert(0, str(ROOT))

from opentpu.host import board as B  # noqa: E402
from opentpu.host import chat as C  # noqa: E402
from opentpu.host import regs as R  # noqa: E402
from opentpu.host import runstate  # noqa: E402
from opentpu.llm import load_spec, model_dir  # noqa: E402
from opentpu.llm import qwen3 as Q  # noqa: E402

COMPUTE = ("sample", "detok", "ui", "status", "compile-wait", "logits-mark")   # timed directly
KNOWN = ["io-write", "args-write", "compile-wait", "prog-upload", "imem-load", "start",
         "counters", "logits-stream", "logits-tail", "logits-mark", "logits-read", "status",
         "sample", "detok", "ui"]


class Profiler:
    """Per-token accounting. `stack` holds the host step being run (the innermost wins); the
    outermost transport operation of a call chain adds its time to [step][critical or
    overlapped] and to ops[op] (calls, bytes, seconds); the COMPUTE steps add their own time."""

    def __init__(self):
        self.on = threading.Event()
        self.t = defaultdict(lambda: [0.0, 0.0])        # item -> [critical, overlapped] s
        self.ops = defaultdict(lambda: [0, 0, 0.0])     # op -> [calls, bytes, seconds]
        self.stack = ["other"]
        self.depth = 0                                  # transport operations in progress
        self.running = False                            # between RUN and HALTED seen
        self.t_halt = self.t_run = None
        self.crit, self.windows = 0.0, 0                # sum of HALTED seen -> next RUN
        self.runs = []                                  # RUN written -> HALTED seen, s
        self.tokens = []                                # per window: {item: critical s}
        self._snap = {}

    def add(self, item, dt, running=None):
        if self.on.is_set():
            self.t[item][int(self.running if running is None else running)] += dt

    def step(self, owner, name, item):
        """Time owner.name as host step `item`."""
        f, prof = getattr(owner, name), self

        def w(*a, **k):
            prof.stack.append(item)
            t0, run0 = time.perf_counter(), prof.running
            try:
                return f(*a, **k)
            finally:
                prof.stack.pop()
                if item in COMPUTE:
                    prof.add(item, time.perf_counter() - t0, run0)
        setattr(owner, name, w)

    def op(self, t, name, op, nbytes=None, after=None):
        """Time transport method t.name as operation `op`; after(args, result) runs last."""
        f, prof = getattr(t, name), self

        def w(*a, **k):
            if threading.current_thread() is not threading.main_thread():
                t0 = time.perf_counter()                # the DMA worker: counted, not filed
                r = f(*a, **k)
                if prof.on.is_set():
                    o = prof.ops[op + " (worker)"]
                    o[0] += 1
                    o[1] += nbytes(a) if nbytes else 0
                    o[2] += time.perf_counter() - t0
                return r
            prof.depth += 1
            t0, run0 = time.perf_counter(), prof.running
            try:
                r = f(*a, **k)
            finally:
                prof.depth -= 1
            dt = time.perf_counter() - t0
            if after is not None:                       # HALTED seen / RUN written: first
                after(a, r)
            if prof.depth == 0 and prof.on.is_set():
                o = prof.ops[op]
                o[0] += 1
                o[1] += nbytes(a) if nbytes else 0
                o[2] += dt
                if run0 and not prof.running:           # HALTED seen inside: split the time
                    prof.add(prof.stack[-1], prof.t_halt - t0, True)
                    prof.add(prof.stack[-1], time.perf_counter() - prof.t_halt, False)
                else:
                    prof.add(prof.stack[-1], dt, run0)
            return r
        setattr(t, name, w)

    def instrument_transport(self, t):
        prof = self

        def run_written(a, r):
            off, val = a[0], a[1]
            if off == R.R_CTRL and val & R.CTRL_RUN:
                now = time.perf_counter()
                if prof.on.is_set() and prof.t_halt is not None:
                    prof.crit += now - prof.t_halt
                    prof.windows += 1
                    cur = {k: v[0] for k, v in prof.t.items()}
                    tok = {k: v - prof._snap.get(k, 0.0) for k, v in cur.items()}
                    tok = {k: v for k, v in tok.items() if v > 0}
                    tok["critical"] = now - prof.t_halt
                    prof.tokens.append(tok)
                    prof._snap = cur
                elif prof.on.is_set():
                    prof._snap = {k: v[0] for k, v in prof.t.items()}
                prof.t_halt, prof.t_run, prof.running = None, now, True

        def status_read(a, r):
            if prof.running and (
                    (a[0] == R.R_STATUS and r & R.ST_HALTED) or           # reg_read
                    (len(a) > 2 and a[0] == R.R_STATUS and a[1] & R.ST_HALTED)):  # poll
                prof.t_halt, prof.running = time.perf_counter(), False
                if prof.on.is_set():
                    prof.runs.append(prof.t_halt - prof.t_run)
        self.op(t, "mem_write", "dma-write", lambda a: len(a[2]))
        self.op(t, "mem_read", "dma-read", lambda a: a[2])
        self.op(t, "reg_write", "reg-write", after=run_written)
        self.op(t, "reg_read", "reg-read", after=status_read)
        self.op(t, "reg_read_many", "reg-read-many")
        self.op(t, "poll", "poll", after=status_read)


def main(argv=None):
    ap = argparse.ArgumentParser(description=__doc__.split("\n")[0])
    ap.add_argument("--model", default="lfm2")
    ap.add_argument("--backend", default="board", choices=["board", "fake"])
    ap.add_argument("--dev", default="/dev/xdma0")
    ap.add_argument("--tokens", type=int, default=64)
    ap.add_argument("--cap", type=int, default=2048)
    ap.add_argument("--prompt", default="Write a short story about a lighthouse keeper.")
    ap.add_argument("--greedy", action="store_true")
    ap.add_argument("--no-stream", action="store_true",
                    help="read the logits after the run (no streamed logits)")
    ap.add_argument("--seed", type=int, default=0)
    ap.add_argument("--fake-ms", type=float, default=80.0)
    ap.add_argument("--fake-stream", action="store_true",
                    help="--backend fake: streamed logits (the fake writes them piece by piece "
                         "in the run's second half)")
    ap.add_argument("--fake-skew", type=float, default=0.0,
                    help="--backend fake: its CYCLES say a run this fraction longer than its "
                         "wall time (a CORE_KHZ off the real clock; negative: shorter)")
    ap.add_argument("--per-position", action="store_true",
                    help="a decode program per position (default: the resident one when the "
                         "bitstream takes run arguments)")
    ap.add_argument("--fake-no-args", action="store_true",
                    help="--backend fake: a bitstream without run arguments (CAPS bit25)")
    ap.add_argument("--json")
    ap.add_argument("--wformat", default="int8", choices=["int8", "fp4", "int4", "mix"],
                    help="weight format of the layers (docs/quant.md)")
    ap.add_argument("--head-format", default=None, choices=["int8", "fp4", "int4"],
                    help="weight format of the LM head (default: --wformat)")
    ap.add_argument("--card-loop", action="store_true",
                    help="the decode loop on the card (Engine.generate_card)")
    a = ap.parse_args(argv)
    from transformers import AutoTokenizer
    path = model_dir(a.model)
    tok = AutoTokenizer.from_pretrained(path)
    spec = load_spec(path)
    P = Profiler()
    wkw = dict(wformat=a.wformat, head_format=a.head_format)
    if a.backend == "board":
        backend, cfg = C.make_backend("board", spec, a.cap, a.dev, path.name)
    else:
        from opentpu.host.fake import FakeTransport
        from opentpu.isasim import board_config
        probe = spec.image(board_config(DRAM_BYTES=1 << 32), a.cap, **wkw,
                           **({"lookup": True} if not a.per_position and Q.has_lookup(spec)
                              else {}))
        ch = 1 << max(20, (probe.nbytes // 2 + (1 << 20)).bit_length())
        tr = FakeTransport(ch_bytes=ch, run_s=a.fake_ms / 1e3,
                           cycles=int(a.fake_ms * (1 + a.fake_skew) * 1e5), devname=None,
                           args=not a.fake_no_args)
        cfg = B.device_config(B.Board(tr, lock=False).info(), DRAM_BYTES=2 * ch)
        backend = lambda c, imgs: B.BoardBackend(c, imgs, transport=tr, model=path.name)  # noqa
    # host steps: their transport operations are filed under them
    P.step(B.BoardBackend, "write", "io-write")
    P.step(B.Board, "load_program", "imem-load")
    P.step(B.Board, "set_args", "args-write")
    P.step(B.Board, "start", "start")
    P.step(B.Board, "wait", "counters")
    P.step(B.BoardBackend, "_stream_logits", "logits-stream")
    P.step(B.BoardBackend, "_stream_tail", "logits-tail")
    P.step(B.BoardBackend, "_settle", "logits-mark")
    P.step(B.BoardBackend, "read", "logits-read")
    P.step(runstate.RunnerStatus, "token", "status")
    P.step(Q.Engine, "_program", "compile-wait")
    P.step(Q.Engine, "_decode", "compile-wait")
    b_write = B.Board.write

    def board_write(self, addr, data):                  # the program's DMA in load_program
        if P.stack[-1] != "imem-load":
            return b_write(self, addr, data)
        P.stack.append("prog-upload")
        try:
            return b_write(self, addr, data)
        finally:
            P.stack.pop()
    B.Board.write = board_write

    eng = Q.Engine(spec, Q.load_weights(path), cap=a.cap, cfg=cfg, backend=backend, **wkw,
                   resident=not a.per_position)
    eng.stream_logits = not a.no_stream
    if a.backend == "fake":         # logits the sampler works on as on real ones (no ties)
        import numpy as np
        lg = np.random.default_rng(0).normal(0, 3, eng.image.v_loc).astype(np.float32)
        eng.backend.board.write(eng.image.io["logits"], lg)
        if a.fake_stream:           # the run writes its logits (all at the halt): streamed
            tr.streams = eng.backend.streams = True
            tr.logits = (eng.image.io["logits"], 4 * eng.image.v_loc,
                         4 * min(Q.HEAD_CHUNK, eng.cfg.TMEM_WORDS // 8))
    khz = (getattr(eng.backend, "info", {}) or {}).get("core_khz") or 100_000
    P.instrument_transport(eng.backend.board.t)
    sp = C.sampling(spec, argparse.Namespace())
    pick = C.sampler(0 if a.greedy else sp["temperature"], sp["top_k"], sp["top_p"], a.seed,
                     sp["repetition_penalty"])
    step0 = {}

    def first():                                        # the first pick: the prefill is done
        if "t" not in step0:
            step0["t"] = time.perf_counter()
            step0["n"] = len(eng.stats)
            P.t_halt = P.t_halt or time.perf_counter()
            P.on.set()

    def timed_pick(logits, ctx=()):
        first()
        t0 = time.perf_counter()
        r = pick(logits, ctx)
        P.add("sample", time.perf_counter() - t0)
        return r

    class TimedStream:                                  # pick.stream, its work timed
        def __init__(self, ctx):
            self.s = pick.stream(ctx)

        def begin(self, n, *x):
            self.s.begin(n, *x)

        def feed(self, lo, v):
            t0 = time.perf_counter()
            self.s.feed(lo, v)
            P.add("sample", time.perf_counter() - t0)

        def result(self):
            first()
            t0 = time.perf_counter()
            r = self.s.result()
            P.add("sample", time.perf_counter() - t0)
            return r
    timed_pick.stream = TimedStream
    timed_pick.warm = pick.warm
    lands, comp = [], [0.0]
    if a.card_loop:                 # Chat.on_card: the picks after the first on the card
        timed_pick.greedy, timed_pick.params, timed_pick.rng = pick.greedy, pick.params, pick.rng
        run_gen, gen_prog = B.BoardBackend.run_generate, Q.Engine._generate_prog

        def run_generate(self, progs, out, n, on_token=None, *x, **k):
            def landed(t):                              # when each of the card's tokens landed
                lands.append(time.perf_counter())
                if on_token is not None:
                    on_token(t)
            return run_gen(self, progs, out, n, landed, *x, **k)

        def generate_prog(self, *x, **k):               # the buckets' compiles (first use)
            t0 = time.perf_counter()
            try:
                return gen_prog(self, *x, **k)
            finally:
                comp[0] += time.perf_counter() - t0
        B.BoardBackend.run_generate, Q.Engine._generate_prog = run_generate, generate_prog
    chat = C.Chat(eng, tok, False, timed_pick, a.tokens, clock_mhz=khz / 1e3)
    if a.card_loop and not chat.on_card:
        sys.exit("--card-loop: this engine / bitstream does not run the decode loop (CAPS bit30)"
                 " or the sampler's settings are not the device's (generate.Sampling)")
    dec = tok.decode

    def decode(*x, **k):
        t0 = time.perf_counter()
        try:
            return dec(*x, **k)
        finally:
            P.add("detok", time.perf_counter() - t0)
    tok.decode = decode
    sink = io.StringIO()

    def upd(delta, turn):
        t0 = time.perf_counter()
        sink.write(delta)
        P.add("ui", time.perf_counter() - t0)
    _, turn = chat.ask(a.prompt, upd)
    P.on.clear()
    wall = time.perf_counter() - step0["t"]
    n = len(eng.stats) - step0["n"]
    cyc = sum(s["cycles"] for s in eng.stats[step0["n"]:])
    if a.card_loop:
        steps = turn.decode_steps
        print(f"{path.name} on {a.backend}: the decode loop on the card, {steps} tokens in {n} "
              f"run(s) after the first; wall {steps / wall:.2f} tok/s, device "
              f"{steps * khz * 1e3 / cyc:.2f} tok/s ({cyc / steps / 1e6:.3f} Mcycles/token), "
              f"wall {100 * (steps / wall) / (steps * khz * 1e3 / cyc) - 100:+.1f}% of device")
        run_s = lands[-1] - lands[0] if len(lands) > 1 else 0.0
        steady = (len(lands) - 1) / run_s if run_s > 0 else 0.0
        start_ms = 1e3 * (lands[0] - step0["t"]) if lands else 0.0
        if steady:
            print(f"{path.name} on {a.backend}: the decode loop on the card from its first token "
                  f"to its last ({len(lands)} tokens): {steady:.2f} tok/s, "
                  f"{100 * steady / (steps * khz * 1e3 / cyc) - 100:+.1f}% of device; before "
                  f"the first: {start_ms:.1f} ms ({1e3 * comp[0]:.1f} ms compiling)")
        if a.json:
            Path(a.json).write_text(json.dumps({
                "model": path.name, "wformat": a.wformat, "head_format": a.head_format,
                "card_loop": True, "steps": steps, "runs": n, "wall_tok_s": steps / wall,
                "dev_tok_s": steps * khz * 1e3 / cyc, "steady_tok_s": steady,
                "start_ms": start_ms, "compile_ms": 1e3 * comp[0],
                "reply_ids": [int(x) for x in chat._reply]}, indent=1))
        eng._drain()
        eng.backend.close()
        return
    dev_ms = 1e3 * cyc / n / (khz * 1e3)
    per = {k: (1e3 * v[0] / n, 1e3 * v[1] / n) for k, v in P.t.items()}
    crit = 1e3 * P.crit / max(P.windows, 1)
    crit_known = sum(per.get(k, (0, 0))[0] for k in KNOWN)
    bid = (getattr(eng.backend, "info", {}) or {}).get("build_id")
    fmt = a.wformat + (f", head {a.head_format}" if a.head_format else "")
    streamed = eng.stream_logits and getattr(eng.backend, "streams", False)
    print(f"{path.name} ({fmt}, {'streamed logits' if streamed else 'no stream'}) on "
          f"{a.backend}" + ("" if bid is None else f" (build {bid:08x})")
          + f": {n} decode steps, prompt fed {step0['n']} tokens")
    print(f"{'ms per token':<15} {'critical':>9} {'overlapped':>11}")
    for k in KNOWN:
        if k in per:
            print(f"{k:<15} {per[k][0]:9.3f} {per[k][1]:11.3f}")
    print(f"{'other':<15} {max(0.0, crit - crit_known):9.3f}")
    over = 1e3 * sum(P.runs) / max(len(P.runs), 1) - dev_ms
    # the mean hides where it comes from (one run seen 14 ms late is 0.15 ms per token of 96)
    over_med = 1e3 * sorted(P.runs)[len(P.runs) // 2] - dev_ms if P.runs else 0.0
    over_max = 1e3 * max(P.runs) - dev_ms if P.runs else 0.0
    if P.tokens:                    # where the outliers are: the worst tokens and their items
        worst = sorted(P.tokens, key=lambda t: -t["critical"])[:3]
        print("slowest tokens (critical ms): " + "; ".join(
            f"{1e3 * t['critical']:.2f} (" + ", ".join(
                f"{k} {1e3 * v:.2f}" for k, v in sorted(
                    ((k, v) for k, v in t.items() if k != "critical"), key=lambda x: -x[1])[:3])
            + ")" for t in worst))
    print(f"host critical path (HALTED seen -> next RUN): {crit:.3f} ms/token over "
          f"{P.windows} tokens; device {dev_ms:.3f} ms/token; HALTED seen {over:.3f} ms after "
          f"the run's end (poll overshoot; median {over_med:.3f}, max {over_max:.3f})")
    print("transport per token: " + ", ".join(
        f"{k} {v[0] / n:.1f}x {v[1] / n / 1024:.1f} KiB {1e3 * v[2] / n:.3f} ms"
        for k, v in sorted(P.ops.items())))
    ls = getattr(eng.backend, "last_stream", None) if streamed else None
    if ls:
        print(f"streamed logits (last token): {ls.get('during')} of {ls.get('pieces')} pieces "
              f"during the run, {ls.get('probes')} probes, then "
              f"{ls.get('tail_bytes', 0) / 1024:.0f} KiB in {1e3 * ls.get('tail_s', 0):.3f} ms")
    # the first window (the prompt's last pick, the first decode program's upload) is the
    # start-up of the decode, not a step of it: the rate from the first decode run on
    first_ms = 1e3 * P.tokens[0]["critical"] if P.tokens else 0.0
    steady = n / (wall - first_ms / 1e3) if wall > first_ms / 1e3 else 0.0
    print(f"wall {n / wall:.2f} tok/s, device {n * khz * 1e3 / cyc:.2f} tok/s "
          f"({cyc / n / 1e6:.3f} Mcycles/token); from the first decode run {steady:.2f} tok/s "
          f"(the first window, {first_ms:.2f} ms, left out)")
    if a.json:
        Path(a.json).write_text(json.dumps({
            "model": path.name, "wformat": a.wformat, "head_format": a.head_format,
            "streamed": streamed, "steps": n, "build_id": bid,
            "ms": {k: {"critical": c, "overlapped": o} for k, (c, o) in per.items()},
            "critical_ms": crit, "device_ms": dev_ms, "overshoot_ms": over,
            "ops": {k: {"calls": v[0] / n, "bytes": v[1] / n, "ms": 1e3 * v[2] / n}
                    for k, v in P.ops.items()},
            "wall_tok_s": n / wall, "dev_tok_s": n * khz * 1e3 / cyc,
            "steady_tok_s": steady, "first_window_ms": first_ms,
            "reply_ids": [int(x) for x in chat._reply],
            # per token (window HALTED seen -> next RUN): the critical ms of each item
            "tokens": [{k: round(1e3 * v, 4) for k, v in t.items()} for t in P.tokens]},
            indent=1))
    eng._drain()
    eng.backend.close()


if __name__ == "__main__":
    main()