Run transformer inference 4.76x faster on Apple Silicon Neural Engine

Direct Neural Engine inference for transformers on Apple Silicon.

Espresso compiles MIL programs straight to ANE silicon through reverse-engineered private APIs (_ANEClient, _ANEInMemoryModel). No CoreML in the hot path. No per-token recompilation. IOSurface buffers and fused multi-layer kernels for Apple Silicon.

• Direct ANE path - private-API compile once, reuse across decode steps
• Fused multi-layer kernels - fewer ANE dispatches per token
• Zero-copy I/O - NEON-vectorized surface reads, vDSP/Metal where useful
• Pure Swift 6.2 core - ~Copyable move-only tensors, strict concurrency, typed throws
• Zero required third-party packages - clean clone resolves and builds on macOS 15+

Product journeys

Pick one path. Everything else is optional or research.

1. Try it (demo)

git clone https://github.com/christopherkarani/Espresso.git
cd Espresso
./espresso doctor   # host readiness check (scripts, ANE, Python)
./espresso prepare  # bootstrap GPT-2 demo weights + tokenizer (network; torch/transformers)
./espresso          # builds if needed, launches the GPT-2 TUI demo

First demo run also bootstraps assets automatically when they are missing. That step needs
network access and a Python with torch + transformers (Espresso can create a managed venv).

2. Serve a model (.esp bundles)

Portable model bundles are the retained serving path:

# Pack a prepared native model directory into a portable bundle
swift run espc pack-native /path/to/model /tmp/model.esp --overwrite

# Inspect / run
swift run esprun inspect /tmp/model.esp
swift run esprun generate /tmp/model.esp "Hello" 32

# Same bundle boundary via the generate CLI
swift run espresso-generate generate --bundle /tmp/model.esp --max-tokens 32 "Hello"

3. Chat with a real open-weight model (Qwen2.5-1.5B-Instruct)

# Convert the local Hugging Face snapshot (or download it) and pack a .esp bundle
python3 scripts/convert_qwen25_05b_to_esp.py --model Qwen/Qwen2.5-1.5B-Instruct

# Multi-turn chat. Fallback is disabled. Live tok/s, TTFT, and J/tok stay in the TUI.
./espresso chat --model ~/Library/Caches/Espresso/qwen25-15b/Qwen2.5-1.5B-Instruct.esp

Qwen2.5-1.5B-Instruct decodes through Espresso's ANE hybrid path: Q/K/V and the SwiGLU
FFN on the Neural Engine; RoPE, attention, and the ~467 MB LM head on the CPU by design
(cpu_fp16_tiled). Chat keeps Qwen Instruct history across turns. Commands: /reset
/retry /exit. Ctrl-C cancels the current completion.

Throughput and energy are live footer measurements for that completion, not README
headlines. Optional dual-pane vs MLX (same checkpoint, greedy, fp16 vs fp16; compile
excluded from tok/s):

./espresso chat --vs mlx --greedy --model ~/Library/Caches/Espresso/qwen25-15b/Qwen2.5-1.5B-Instruct.esp

See docs/qwen15b-parity.md for the 1.5B greedy contract and
the commands that regenerate it.

4. Reproduce 0.5B greedy parity

# Default converter target remains Qwen2.5-0.5B-Instruct
python3 scripts/convert_qwen25_05b_to_esp.py

ESPRESSO_REALMODEL_DISABLE_HYBRID_FALLBACK=1 \
  ./espresso generate --model ~/Library/Caches/Espresso/qwen25-05b/Qwen2.5-0.5B-Instruct.esp \
  -n 24 "The capital of France is"

Qwen2.5-0.5B-Instruct is the parity/repro path. It reproduces a PyTorch fp32 reference
on a fixed 12-prompt greedy suite: 10 of 12 sequences match token-for-token, 341 of
384 tokens agree. That 10/12 is generate-path evidence (cpu_fp16_tiled LM head). A
separate probe - chained per-layer hidden states plus a NumPy/Python LM head, not the
served tiled classifier - agrees with PyTorch to 9.3e-5 in logits on the fp32 CPU
stack through all 24 layers, and to ~0.96 on the ANE hybrid stack. The two greedy
divergences come from fp16 execution on the ANE (up to ~1 logit of error) and both land
on precisely the reference's runner-up token at top-1/top-2 gaps of 0.027 and 0.069.
That is one model, greedy, fp16, measured - not a general-model claim, and not a speed
claim.
See docs/qwen-parity.md for the per-layer report, the exact
commands, and every ANE limitation hit along the way.

5. Embed the library (ANEKernel)

// Package.swift
.package(url: "https://github.com/christopherkarani/Espresso.git", from: "0.9.0")

import ANERuntime

let kernel = try ANEKernel(
    milText: myMIL,
    weights: blobs,
    inputSizes: [input],
    outputSizes: [output]
)
try kernel.eval()                         // runs on Neural Engine
let result = kernel.outputSurface(at: 0)  // zero-copy read

For end-to-end text generation from prepared weights, use RealModelInference or a .esp bundle rather than hand-writing MIL.

./espresso install                            # PATH shim → this checkout (does not download weights)
./espresso prepare                            # download/convert GPT-2 demo assets
./espresso compare --no-power "Hello"         # side-by-side vs CoreML (demo weights)
swift run espresso-bench --ane-only --inference --layers 6
swift run espresso-train                      # experimental ANE training loop

Benchmark

Numbers below match the checked-in machine-readable results in
benchmarks/results/latest.json
(M3 Max, macOS 15.0, Espresso 1.1.0).

Espresso vs CoreML (local 6-layer Stories artifact)

All Espresso / CoreML numbers: 6-layer local artifact · dim=768 · 12 heads · 32k vocab · seqLen=256 · M3 Max · macOS 15.
This is a research / demo artifact family, not a pretrained production model.

What these numbers are not

• Not a claim about full GPT-2 117M or llama.cpp Metal on the same workload
• Not every serving path: retained exact hybrid .esp Stories runs can land lower in wall-clock tok/s after compile and full decode accounting
• Not trunk-only or partial-pipeline peaks from blog posts - only figures backed by latest.json (or a PR artifact from the reproduce script) are project claims

RESULTS_DIR=results/$(date +%Y%m%d-%H%M%S) \
REPEATS=5 WARMUP=3 ITERATIONS=20 \
./scripts/reproduce_local_real_artifact_claim.sh

Platform Compatibility

macOS 15+ required. iOS / tvOS not supported out of the box (private API entitlements differ per platform).

How It Works

                    ┌─────────────────────┐
                    │   MIL Program Text   │  Generated per-kernel
                    └──────────┬──────────┘
                               ▼
                    ┌─────────────────────┐
                    │  _ANEClient compile  │  Private API (dlopen)
                    └──────────┬──────────┘
                               ▼
                    ┌─────────────────────┐
                    │    ANE E5 Binary     │  Cached by system
                    └──────────┬──────────┘
                               ▼
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
     ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
     │  IOSurface   │ │  IOSurface   │ │  IOSurface   │
     │   (input)    │ │  (weights)   │ │  (output)    │
     └──────┬───────┘ └──────────────┘ └──────┬───────┘
            │          ANE Hardware            │
            └──────────────eval───────────────┘

The decode loop compiles once and reuses the program across all steps. KV cache lives in IOSurface buffers - not marshaled through CoreML.

Architecture

ANEInterop (ObjC/C — private API bridge)
  └── ANETypes (~Copyable value types, IOSurface I/O)
          ├── MILGenerator (kernel variants)
          │       └── ANERuntime (compile, eval, surface management)
          │               └── Espresso / RealModelInference (serving, decode)
          │                       ├── espresso-generate / esprun (CLI)
          │                       └── ESPBundle (portable .esp)
          └── CPUOps (Accelerate/vDSP kernels)

SPM Integration

// Package.swift
dependencies: [
    .package(url: "https://github.com/christopherkarani/Espresso.git", from: "0.9.0")
],
targets: [
    .target(name: "MyApp", dependencies: [
        .product(name: "ANERuntime", package: "Espresso"),
        .product(name: "ANETypes",   package: "Espresso"),
    ])
]

import ANERuntime
import ANETypes

let kernel = try ANEKernel(
    milText: milText,
    weights: weightBlobs,
    inputSizes: [inputByteSize],
    outputSizes: [outputByteSize]
)
try kernel.eval()
let output = try kernel.outputSurface(at: 0)

Dependencies

Zero third-party Swift packages. The package graph depends only on Apple system frameworks (Foundation, Accelerate, IOSurface, Metal, CoreML). A clean clone of this repo alone must resolve and build.

Requirements

Testing

swift test                                                    # unit tests (no ANE needed)
ANE_HARDWARE_TESTS=1 swift test --filter "ANERuntimeTests|EspressoTests"  # hardware tests
OBJC_CROSS_VALIDATION=1 ANE_HARDWARE_TESTS=1 swift test --filter CrossValidationTests  # parity

CI runs non-hardware unit tests including ESP bundle/runtime and RealModelInference unit suites, asserts a zero-dependency default graph, and checks README claim numbers against latest.json. Hardware ANE tests run on self-hosted matrix jobs.

Research vs product

Do not treat research/ numbers or flags as supported product behavior.