Run transformer inference 4.76x faster on Apple Silicon Neural Engine

Direct Neural Engine inference for transformers on Apple Silicon.

Espresso compiles MIL programs straight to ANE silicon through reverse-engineered private APIs (_ANEClient, _ANEInMemoryModel). No CoreML in the hot path. No per-token recompilation. IOSurface buffers and fused multi-layer kernels for Apple Silicon.

• Direct ANE path - private-API compile once, reuse across decode steps
• Fused multi-layer kernels - fewer ANE dispatches per token
• Zero-copy I/O - NEON-vectorized surface reads, vDSP/Metal where useful
• Pure Swift 6.2 core - ~Copyable move-only tensors, strict concurrency, typed throws
• Zero required third-party packages - clean clone resolves and builds on macOS 15+

Product journeys

Pick one path. Everything else is optional or research.

1. Try it (demo)

git clone https://github.com/christopherkarani/Espresso.git
cd Espresso
./espresso          # builds, downloads demo weights, launches TUI
./espresso doctor   # host readiness check

2. Serve a model (.esp bundles)

Portable model bundles are the retained serving path:

# Pack a prepared native model directory into a portable bundle
swift run espc pack-native /path/to/model /tmp/model.esp --overwrite

# Inspect / run
swift run esprun inspect /tmp/model.esp
swift run esprun generate /tmp/model.esp "Hello" 32

# Same bundle boundary via the generate CLI
swift run espresso-generate generate --bundle /tmp/model.esp --max-tokens 32 "Hello"

3. Embed the library (ANEKernel)

// Package.swift
.package(url: "https://github.com/christopherkarani/Espresso.git", from: "1.0.0")

import ANERuntime

let kernel = try ANEKernel(
    milText: myMIL,
    weights: blobs,
    inputSizes: [input],
    outputSizes: [output]
)
try kernel.eval()                         // runs on Neural Engine
let result = kernel.outputSurface(at: 0)  // zero-copy read

For end-to-end text generation from prepared weights, use RealModelInference or a .esp bundle rather than hand-writing MIL.

./espresso install                            # shim → ~/.local/bin
./espresso compare --no-power "Hello"         # side-by-side vs CoreML (demo weights)
swift run espresso-bench --ane-only --inference --layers 6
swift run espresso-train                      # experimental ANE training loop

Benchmark

Numbers below match the checked-in machine-readable results in
benchmarks/results/latest.json
(M3 Max, macOS 15.0, Espresso 1.1.0).

Espresso vs CoreML (local 6-layer Stories artifact)

All Espresso / CoreML numbers: 6-layer local artifact · dim=768 · 12 heads · 32k vocab · seqLen=256 · M3 Max · macOS 15.
This is a research / demo artifact family, not a pretrained production model.

What these numbers are not

• Not a claim about full GPT-2 117M or llama.cpp Metal on the same workload
• Not every serving path: retained exact hybrid .esp Stories runs can land lower in wall-clock tok/s after compile and full decode accounting
• Not trunk-only or partial-pipeline peaks from blog posts - only figures backed by latest.json (or a PR artifact from the reproduce script) are project claims

RESULTS_DIR=results/$(date +%Y%m%d-%H%M%S) \
REPEATS=5 WARMUP=3 ITERATIONS=20 \
./scripts/reproduce_local_real_artifact_claim.sh

Platform Compatibility

macOS 15+ required. iOS / tvOS not supported out of the box (private API entitlements differ per platform).

How It Works

                    ┌─────────────────────┐
                    │   MIL Program Text   │  Generated per-kernel
                    └──────────┬──────────┘
                               ▼
                    ┌─────────────────────┐
                    │  _ANEClient compile  │  Private API (dlopen)
                    └──────────┬──────────┘
                               ▼
                    ┌─────────────────────┐
                    │    ANE E5 Binary     │  Cached by system
                    └──────────┬──────────┘
                               ▼
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
     ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
     │  IOSurface   │ │  IOSurface   │ │  IOSurface   │
     │   (input)    │ │  (weights)   │ │  (output)    │
     └──────┬───────┘ └──────────────┘ └──────┬───────┘
            │          ANE Hardware            │
            └──────────────eval───────────────┘

The decode loop compiles once and reuses the program across all steps. KV cache lives in IOSurface buffers - not marshaled through CoreML.

Architecture

ANEInterop (ObjC/C — private API bridge)
  └── ANETypes (~Copyable value types, IOSurface I/O)
          ├── MILGenerator (kernel variants)
          │       └── ANERuntime (compile, eval, surface management)
          │               └── Espresso / RealModelInference (serving, decode)
          │                       ├── espresso-generate / esprun (CLI)
          │                       └── ESPBundle (portable .esp)
          └── CPUOps (Accelerate/vDSP kernels)

SPM Integration

// Package.swift
dependencies: [
    .package(url: "https://github.com/christopherkarani/Espresso.git", from: "1.0.0")
],
targets: [
    .target(name: "MyApp", dependencies: [
        .product(name: "ANERuntime", package: "Espresso"),
        .product(name: "ANETypes",   package: "Espresso"),
    ])
]

import ANERuntime
import ANETypes

let kernel = try ANEKernel(
    milText: milText,
    weights: weightBlobs,
    inputSizes: [inputByteSize],
    outputSizes: [outputByteSize]
)
try kernel.eval()
let output = try kernel.outputSurface(at: 0)

Dependencies

Zero third-party Swift packages. The package graph depends only on Apple system frameworks (Foundation, Accelerate, IOSurface, Metal, CoreML). A clean clone of this repo alone must resolve and build.

Requirements

Testing

swift test                                                    # unit tests (no ANE needed)
ANE_HARDWARE_TESTS=1 swift test --filter "ANERuntimeTests|EspressoTests"  # hardware tests
OBJC_CROSS_VALIDATION=1 ANE_HARDWARE_TESTS=1 swift test --filter CrossValidationTests  # parity

CI runs non-hardware unit tests including ESP bundle/runtime and RealModelInference unit suites, asserts a zero-dependency default graph, and checks README claim numbers against latest.json. Hardware ANE tests run on self-hosted matrix jobs.

Research vs product

Do not treat research/ numbers or flags as supported product behavior.