Running a Model Locally and Talking to It: Runtimes, Speed, and the Gateway in Between

Running a model on your own hardware is usually framed as a cost decision, and that framing is where most local setups go wrong. The bill is the least interesting part. What you actually get is a request path you own end to end: nothing leaves the machine unless you route it out, the model does not change under you between Tuesday and Wednesday, and the latency floor is set by your silicon rather than by someone else's queue.

What you also get is three new problems the API never made you think about. Something has to serve the model. Something has to make it fit — a 26-billion-parameter model does not care that your laptop has 8GB. And something has to sit between the model and whatever is calling it, because the moment an agent can reach a local endpoint it can usually reach everything else on that machine too.

The nine tools below are ordered by those three problems, plus a fourth that gets skipped almost universally: proving the setup is doing what you think it is. Star counts, licences and last-push dates are in the table below, measured on the date shown, because "is this maintained, and may I legally use it" is the question the READMEs answer least well — one entry here has no licence at all, and another went from 260 to over 6,000 stars in the three weeks before this was published.

Some links to project repositories carry a referral parameter. It costs you nothing, changes nothing about what appears here, and the ordering below is by problem area, not by whether a link earns anything.

The nine tools at a glance

Star counts, languages, licences and last-push dates were read from the GitHub API on 2026-08-22.

Tool	What it does	Language	License	Last push	★
mistral.rs	Inference engine with OpenAI- and Anthropic-compatible endpoints	Rust	MIT	2026-08-22	7620
turbo-fieldfare	Gemma 4 26B in ~2GB RAM on Apple Silicon, experts streamed off SSD	Swift	Apache-2.0	2026-08-22	6264
onecli	Gateway handing agents placeholder credentials, real key injected on egress	TypeScript	Apache-2.0	2026-08-19	3366
clawpatrol	Firewall on agent traffic: HCL/CEL rules, deny or hold-for-human	Go	MIT	2026-08-19	1019
KVarN	KV-cache quantization as a vLLM backend: 3-5× capacity at FP16 accuracy	Python	Apache-2.0	2026-06-22	466
Espresso	Compiles to the Apple Neural Engine bypassing CoreML, 4.76× on reference	Swift	MIT	2026-08-22	166
MemStitch	Shares KV-cache blocks between agents instead of re-running prefill	Python	none	2026-07-25	22
picchio	Checks the GPU actually computed: prefill / decode / wallclock split	Python	MIT	2026-08-22	17
libargus.cc	LLM + STT + TTS + vision in one native runtime for the JVM	Java	MIT	2026-08-22	15

What counts as local here

Anything that runs model weights on hardware you control, or that governs the path to it. That covers four mechanisms and deliberately excludes a fifth.

It includes serving (a process that loads weights and answers requests), fitting (making a model run on memory it nominally does not fit in), verifying (proving the hardware you paid for is the hardware doing the work), and access (deciding who and what may reach the endpoint, with which credentials).

It excludes spending less on an API, which is a different problem with a different toolchain, covered separately in our LLM cost hub. The line matters because the two get conflated constantly: a cheaper cloud model and a local model both reduce a bill, but only one of them changes who holds your data, and only one of them makes your throughput a hardware question. A tool that routes prompts to a smaller vendor model is a cost tool. A tool that streams expert weights off an SSD is a local-inference tool. Mixing them produces a list that helps with neither.

Start with a runtime, not with a benchmark

The order here matters less than in the cost hub, but one sequencing mistake is common enough to name. People start by optimising — quantization format, KV-cache settings, an ANE backend — before they have a plain working runtime and a measured baseline. Then the optimisation shows a number, and there is nothing to compare it against.

Get one runtime answering requests at a known speed. Measure it, including whether the GPU was involved at all. Only then reach for the tools that make it fit better or go faster, and change one thing at a time.

Serving the model

mistral.rs is the general-purpose answer in this list: a Rust inference engine that runs Hugging Face models with no configuration — architecture, quantization format and chat template are auto-detected — and serves both OpenAI-compatible /v1 endpoints and Anthropic-compatible Messages endpoints from the same process. It handles text, vision, audio and speech in one engine, exposes Prometheus metrics, and ships an MCP client so the local model can call external tools. If you want one process that most existing clients can point at unchanged, start here.

libargus.cc solves the same problem for a narrower audience and does it more thoroughly: a zero-allocation native runtime consolidating text generation, Whisper speech-to-text, text-to-speech and multimodal encoding into one process-global backend built on GGML and llama.cpp, with a C API designed for JVM calls through Project Panama's foreign-function interface. One loaded model backs multiple concurrent sessions, and hot paths avoid heap allocation entirely. At 15 stars it is early — but if your application is a JVM service, the alternative is usually four subprocesses and a lot of copying.

Making it fit

turbo-fieldfare is the most striking entry here: Gemma 4 26B — 26 billion total parameters — running in roughly 2GB of RAM on any Apple Silicon Mac, including 8GB machines, by keeping only the shared core and KV cache resident and streaming the mixture-of-experts weights each token needs directly off the SSD. It is a purpose-built Swift and Metal runtime rather than a wrapper, and the project publishes 102 measured experiments describing how the design was arrived at. Model-specific by construction, which is the trade: it is not a general runtime and does not pretend to be.

Espresso goes at Apple hardware from the other end, compiling model programs straight onto the Neural Engine through private APIs and bypassing CoreML entirely — 1.08 ms/token against CoreML's 5.09 ms/token on the project's 6-layer reference model, with the KV cache living in IOSurface buffers instead of being marshalled across a framework boundary. It also supports training on the ANE, which almost nothing else does. The caveat is structural and worth stating plainly: private APIs are not a contract, and an OS update can end this without warning.

KVarN attacks the memory ceiling on the server side, as a vLLM attention backend that quantizes the KV cache tile by tile — 4-bit keys, 2-bit values in the shipped preset — reporting 3-5× more cache capacity at FP16-level accuracy, calibration-free, behind a single flag. If your bottleneck is concurrent long-context requests rather than model size, this is the lever.

MemStitch is the narrowest idea and the most speculative: when several agents read the same long document in sequence, each normally repeats the whole expensive prefill. MemStitch fingerprints prompt blocks and maps a second agent's attention table onto the first agent's already-computed GPU blocks, skipping prefill rather than copying the cache, with an authorization gate controlling which sessions may stitch into which. The benchmark numbers are the project's own. The repository ships no licence, which means the default legal position is all rights reserved — read it for the idea, and ask the author before building on it.

Proving the GPU actually did the work

picchio is one dependency-free Python file that benchmarks llama.cpp and ollama and answers a question a single tokens-per-second figure hides: it splits prefill, decode and wallclock, then reads the operating system's own GPU meter — ioreg and powermetrics on macOS, NVML on NVIDIA Linux — and compares it against the engine's log, catching the case where a build silently fell back to CPU while claiming full offload. It also walks a GGUF's tensor table to check that the quant in the filename is the quant on disk. Seventeen stars, and the only tool here that can tell you your fast setup was never fast.

Reaching the model safely

onecli sits between agents and the services they call, holding real credentials encrypted at rest and handing agents placeholders instead. When a request passes through, the gateway matches it by host and path, decrypts the right credential and injects it before the call leaves — so the agent's code, logs and context never contain the real key. It runs single-user locally with no login, adds OAuth for teams, and can pull from an external password manager rather than storing secrets itself.

clawpatrol governs the same path by action rather than by credential: a firewall that parses agent traffic at the wire level and evaluates each request against rules you write — SQL verbs and table names for Postgres and ClickHouse, resource and verb and namespace for Kubernetes, method and path and body for HTTP. A rule can deny outright, or hold an action until a human approves it before it reaches the cluster. The distinction from onecli is worth being precise about: one decides whether the agent has the key, the other decides whether this specific action is allowed to happen. Most setups that need either eventually need both.

Doing this natively

The tools above sit outside the assistant. Several of the same problems have install-able answers inside it:

• Local LLM Expert — hardware sizing, quantization formats, inference engines and exact chat templates, in one skill
• Hugging Face Local Models — finding quants, launching llama.cpp servers, converting weights to GGUF
• Semantic Kernel — Local Models — running local models via Foundry Local, LM Studio, Ollama and ONNX
• Multi-Model Advisor — MCP connector that queries several local Ollama models on one question and combines the answers
• AI Safety Guardrails — input/output classifiers and prompt-injection detection, the software-side counterpart to a network firewall

FAQ

Is running a model locally actually cheaper?
Sometimes, and it is the wrong question to lead with. Hardware you already own makes marginal tokens close to free; hardware you buy for this rarely pays back against current API prices at hobby volume. What local buys reliably is control — of data, of model version, of availability — and those are worth stating as the goal, because a project justified purely on cost tends to get cancelled the first time a vendor cuts prices.

Why does my local model feel slower than the same model in the cloud?
Three usual causes, in order of frequency. The GPU is not being used at all and the build silently fell back to CPU — this is common enough that a tool in this list exists mostly to detect it. Prefill and decode are being conflated, so a long prompt reads as a slow model. Or the quantization on disk is not the quantization in the filename. All three are measurable before they are worth theorising about.

Quantization, expert streaming, KV-cache compression — which do I need?
They solve different shortages. Quantization shrinks the weights. Expert streaming keeps most of the weights off memory entirely and pays in SSD reads. KV-cache compression addresses long contexts and concurrency, not model size. Diagnose which resource you are actually out of first; applying the wrong one produces a smaller model that is still too slow.

Do I need a gateway if the model runs on my own machine?
The model being local does not make the agent trustworthy. A local endpoint usually sits on a host that can also reach your database, your cluster and your API keys, and an agent that can call one can generally call the others. A gateway is what makes "may this agent do this" a written rule instead of a hope.

Should I use a project with no licence?
Not as a dependency. No licence means all rights reserved by default — not public domain, and not "probably fine" — so shipping it in a product is a legal exposure regardless of how good the code is. Reading it for the idea is fine, and asking the author to add a licence takes one issue. One entry in the table above is in exactly this position, which is why the column is there.