picchio

One executable Python file that measures local LLMs and leaves reproducible evidence.

Install · Commands · Agent traces · Quant · Lanes · Measured · Examples

Most GPU speed claims are one tok/s number. That number can be
correct and still tell you the wrong story. Three failure modes,
each one command:

• Four quantizations of the same Qwen3.5-9B, all labeled Q4_K_M,
  measure 5.02, 5.02, 5.07 and 5.27 bits per weight
  (the quant label).
• Losing the GPU cost prefill 22x and decode under 2x on the same
  model and file (three lanes).
• The 36 tok/s I remembered from bare llama.cpp reproduced in no
  cell of a 32 cell matrix (silent CPU fallback).

picchio splits prefill, decode and wallclock, reads the engine's
log against the OS's GPU meter, and prints a verdict that says
whether the GPU did the work, and why.

Install

curl -fsSL https://raw.githubusercontent.com/logxio/picchio/main/public/picchio.pyz -o picchio
chmod +x picchio
./picchio

With no arguments it finds your models (ollama tags, the current
folder, the HF and LM Studio caches) and runs the one you pick. A
.gguf path gets the full llama.cpp diagnosis; an ollama tag gets
measurement mode.

Needs python3 and either llama.cpp or ollama. Three passes with a
fixed prompt, the first one cold.
About a minute here with the GPU engaged, a few minutes on CPU. It
writes one cache file under ~/.cache/picchio and nothing else.

./picchio --selftest runs the bundled queue/parity recovery tests. A
repository checkout also replays the raw engine logs in
examples/raw/ and must reproduce every committed
verdict block line for line; the badge runs it on every push. The
downloaded zipapp is one file; its source stays modular under src/.

Commands

In the table, picchio stands for ./picchio.

watch runs next to real work, launching, unloading and signaling
nothing; --for is the sampling window in seconds, and positional
ollama names the loaded model being judged (--engine ollama remains
an alias). With no --for, a bare watch samples 6 s, but watch PID
runs until that process exits (capped at an hour), so pass --for for
a daemon you do not want to wait on:

./picchio watch ollama --for 8 --json --keep-logs evidence/

With --json, the human conclusion stays on stderr and stdout is one
picchio.watch.v1 object, ready for json.loads. --keep-logs writes
watch.samples.jsonl and watch.summary.json; every sample carries a
monotonic timestamp, GPU utilization, power and memory, with unavailable
fields recorded as null and explained in warnings. These are whole-GPU
measurements, never per-PID attribution.

--passes N       measurement passes, first one cold (default 3, min 2)
--keep-logs DIR  save each pass's raw engine output into DIR, plus
                 the sampled GPU curve (telemetry.json) on macOS
                 and on NVIDIA Linux
--no-telemetry   skip the OS-side GPU sampling; the os line then
                 says the verdict rests on engine+timing only
--json           JSON only on stdout; human verdict stays on stderr
--bin PATH       llama.cpp binary to use; prefer llama-completion, the
                 one-shot binary, not the interactive llama-cli
--selftest       verify bundled logic/recovery; a clone also replays raw logs
--version        print version and measurement protocol

Anything after a bare -- goes straight to the llama.cpp binary.
Color only on a terminal (NO_COLOR respected); piped output is
plain ASCII.

Exit codes, for scripting: 0 healthy or no evidence, 2 could not
run, 3 partial offload, 4 silent CPU fallback, 5 conflicting
evidence. guard passes the wrapped command's own exit code through
(128 plus the signal number if it died by one); compare exits 0
once both blocks parse; verify exits 0 when a block is
self-consistent, 5 when its sources fight; watch exits 0 when the
GPU is working, 4 when it sits idle; monitor exits 0 when every
probe held the GPU, 4 when any probe caught a fallback.

run uses the same numeric channel with job-specific meanings: 0 completed,
2 could not run or incomplete, 3 runtime failure, 4 quality failure, 5 causal
evidence conflict, 6 safety stop, 130 interrupted. Runtime success never
upgrades quality success.

Long runs

./picchio run suite.json

That command creates suite.picchio-run/ before work starts. It appends raw
evidence, atomically commits each result, and updates a checkpoint. Run the
same command after interruption: completed cases are skipped and unfinished
cases get a new attempt without overwriting the old one.

Queue artifacts separate runtime and quality, keep per-request input/output,
server log references, memory/swap snapshots, whole-GPU JSONL and hourly
throughput/thermal rollups. Parity runs the same fixtures through bare and
product adapters. It emits a causal verdict only when model hash, runtime build,
context, KV, sampling, task bytes, actual engine wire bytes, cache state and
received process evidence all match. Parity is interleaved by default; failed
causal evidence says DIRECTIONAL and exits 5.

Agent traces

A command adapter can return a generic picchio.agent-trace.v1. Picchio keeps
every round's real engine body, separates the last request's current context
from cumulative prompt work, and writes both JSON and a neutral Markdown table.
Component names are fixed (system instructions, document context, tool schemas, and the rest), never copied from an application's branding. If an app
reports that it compacted below the threshold, the result says FALSE POSITIVE
instead of leaving a cumulative token number open to interpretation. See the
rendered trace and complete adapter
contract. Exact wire bodies stay in the local artifact
directory and may contain private prompt or document content; redact them before
sharing. The Markdown table contains counts and neutral categories only.

The quant label

picchio id MODEL walks the gguf tensor table and prices every
tensor by its ggml type. Our own Q4_K_M measures 5.07 bits per
weight, 27% over the 4 in the name: a mix of five tensor types
from 4.50 to 32.00 bits, and the header's own byte offsets have to
audit to the same total before the card prints. The same Qwen3.5-9B
under the same Q4_K_M label measures 5.02, 5.02, 5.07 and 5.27 bits
per weight across four quantizers, on the 427 tensors all four
files share (examples/quantizers/). The
label does not even promise the same tensor set: one quantizer
ships a 243M-parameter MTP head inside the main file at q8_0,
another ships the same head as a separate repo. The KV cache dtype is
not in the file; the card cites only a concrete marker from the last
measured run for that model and engine (llama.cpp stderr or a readable
local Ollama runner log), and says not recorded when neither exposes
one. On a mixture of experts it reports how many experts wake per token
(examples/id-35b.txt reads 8 of 256, about
3.5B of 34.7B weights per token). Works on a .gguf path or
an ollama tag, read only, exit 0.

Three lanes

Prefill (elsewhere called prompt processing or pp) is
how fast the model reads your prompt; decode (tg or eval) is how
fast it writes the answer; wallclock is generated tokens divided by
everything, load and warmup included.

The lanes fail separately; the chart is two real runs from
examples/, 4 of 10 cpu threads on the CPU side.
Prefill sets the time to first token on a long prompt. A Mac
screenshot showing 500 tok/s is almost always prefill.

Silent CPU fallback

Same machine, same model, same file, forced to CPU
(examples/cpu-fallback.txt):

The WHY line names the first cause the run's own evidence can
prove, or says unknown.

While measuring local models for an app I am building, weeks of
it, bare llama.cpp gave me 36 tok/s and the same model through the
app gave 11.5: that gap is why this repo exists. A 32 cell matrix
across CPU and GPU, cold and warm, reproduced the 36 in no cell, a
rate from a different lane remembered as generation speed. What
the matrix did surface was this silent fallback.

The os line

While the passes run, a background thread reads the OS's own GPU
meter: on macOS, ioreg at 4 Hz plus the powermetrics energy
counters, minus the sudo; on NVIDIA Linux, the driver's NVML. That
is the os line. A full offload claim over a GPU the OS saw stay
flat is CONFLICTING EVIDENCE (exit 5). A build that prints no gpu
evidence while the meter watches the gpu stay idle is SILENT CPU
FALLBACK (exit 4), measured on a real mis-built binary. A missing
source abstains; the line says which evidence is left.

llama-bench

llama-bench answers a different question. Steady state pp and tg
for this machine and model, measured here, same model, same day:

The rented 4090 does the same. Its CUDA build keeps CUDA in that
column at -ngl 0. The 21x prompt side collapse is the CPU run's
only visible trace; there is no load time, no cold/warm split, no
verdict.

Measured

Apple M5, 32 GB, macOS 26.5.1, llama.cpp build 9430 and ollama
0.31.1, roughly 730 prompt tokens and 128 generated tokens per pass,
three passes, the first one cold. That protocol is named in every
block footer (mp1); if it ever changes the tag changes. The lane
columns hold warm medians; the raw engine output behind the first
three rows and the 4090 row is in examples/raw/,
written by --keep-logs.

Run picchio once and paste the verdict block into an issue; a
boring HEALTHY on hardware I do not have is still a data point. A
wrong verdict is the issue I want most.
Misdiagnosis reports
go to the top of the pile.

The 35B result is mostly a load-time problem. 13 of the first
pass's 19 seconds went to reading 20.6 GiB of weights. The
3B-active MoE still decodes 1.6x faster than the dense 9B.

Limits

• Tested: one Apple Silicon machine (llama.cpp and ollama) plus
  one rented Linux RTX 4090 (CUDA). ollama on Linux and Vulkan
  parsing have not touched real hardware; if you run those, I want
  the verdict block either way.
• The full verdict block is llama.cpp and ollama only. MLX, LM
  Studio and other engines get placement truth through watch,
  not the lane table.
• Ollama does not expose per layer placement, device init logs, or
  thread configuration. Placement comes from the memory split it
  reports, unknown when there is none.
• Server mode forces a full prompt read on every pass; on a remote
  url, wallclock includes the network round trip.
• Warm numbers drift between sessions: the 9B medians in this repo
  moved 5 to 8% between two recording rounds on an idle machine.
  --passes 5 tightens a single reading.
• The os meter counts the whole GPU (index 0 on Linux), so it only
  judges runs that started from an idle GPU.