Route LLM requests to cost-optimal models with policy control
Foreman is a self-hosted LLM gateway that routes coding-agent traffic to the cheapest allowed model per policy, tracking every dollar spent.
Why it matters
Control AI coding costs by routing every LLM request through a self-hosted gateway that automatically selects the cheapest model allowed by your policy, tracks spend per conversation, and maintains prompt cache efficiency while keeping all traffic and API keys inside your network.
Outcomes
What it gets done
Route coding agent requests to frontier models for planning and cheap models for routine work based on task type headers
Pin conversations to the same model while prompt cache is warm to avoid burning cache for marginal savings
Track every dollar spent with per-request ledger entries showing measured tokens and actual costs by model and provider
Fail over to alternative providers when circuit breakers trip and explain every routing decision with trace IDs
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/northwood-systems-foreman | bash Overview
Foreman
Foreman is a self-hosted LLM gateway (one Go binary) that routes coding-agent requests to the cheapest policy-allowed model per tier, keeps conversations pinned to a warm prompt cache, and records every request's measured cost and routing reason in a queryable ledger. Use it to cut LLM spend across coding agents by routing routine work to cheaper models while reserving frontier models for planning or high-risk tasks, with real cost tracking rather than estimates; it never inspects message content to make routing decisions.
What it does
Foreman is a self-hosted LLM gateway: a single Go binary that sits between your coding agent and any number of LLM providers, keeping keys and traffic inside your own network. It routes each request to the cheapest model your policy allows, tracks every dollar of spend, and lets you switch models or providers from inside your existing harness with no code changes - keeping frontier models for work that needs them and routing routine work to cheaper ones, governed by policy you write.
Routing follows a fixed precedence: policy floors (high-risk or planning work always gets frontier, regardless of anything else), then explicit X-Foreman-Task-Type/X-Foreman-Risk headers for harnesses that want precise control, then the model the client actually asked for (so an agent's native model picker doubles as a tier switch) - Foreman advertises tier aliases like foreman-plan/foreman-code/foreman-fast via /v1/models and also resolves catalog and upstream model names. Message content itself is never inspected. Within a chosen tier, the router picks the cheapest currently healthy model, since a tier is a list of models across providers that compete on price and cover for each other. With X-Foreman-Session-ID, a conversation sticks to its model and provider while that provider's prompt cache is warm (TTL pulled from the catalog), so routing never burns cache to save a few cents - only an explicit signal (a model-picker switch or a new task-type header) releases a warm route. Provider errors trip a circuit breaker (threshold, cooldown, half-open trials) and retry on the next allowed route.
Every request writes a ledger event (measured tokens, measured cost) and a route trace recording the reason for the decision and every alternative it rejected - queryable via foreman trace <id> or the response's X-Foreman-Trace-ID/X-Foreman-Model/X-Foreman-Provider/X-Foreman-Route-Reason headers. Passthrough is faithful for same-protocol traffic (tools, images, and unmodeled parameters forwarded byte-identical); for cross-protocol traffic (e.g. Anthropic-format clients against OpenAI-compatible providers), the full tool protocol - definitions, results, streamed tool calls - is translated, so Claude Code can run against Qwen- or GLM-class upstreams, while genuinely untranslatable content (images, documents) is refused with an explicit 400 rather than silently dropped.
When to use - and when NOT to
Use Foreman when you're running one or more AI coding agents (Claude Code, Codex, OpenCode, Pi) against paid LLM APIs and want to cut cost by routing routine work to cheaper models while keeping frontier models for planning or high-risk work, all without touching your agent's code - and when you want a real, measured cost ledger (not an estimate) to see exactly where the money goes, per team or per model. It's also a fit for building a "decomposition" pattern on top: breaking large jobs into small subtasks executed by cheap models, verified deterministically, and escalated to frontier models only on failure, since every subtask call still goes through the gateway's policy and spend accounting.
It's not useful if you have no interest in multi-model routing or cost tracking, or if you need it to make routing decisions based on message content - Foreman explicitly never inspects request content, so all routing is driven by explicit headers, policy floors, and the client's own model choice. It also doesn't publish a savings figure for you; it's the instrument that computes yours from your own measured traffic, not a marketing calculator.
Inputs and outputs
Install via Homebrew (brew install northwood-systems/tap/foreman) or go install/prebuilt release/source build, then run interactive setup and start the server:
foreman init
foreman serve
foreman init never asks for a secret directly - provider API keys stay in your own environment variables, referenced in config as ${VAR} and resolved at startup. Configuration is two JSON files plus environment variables: a policy file defining API key hashes and which models count as cheap/mid/frontier and what task types each role may handle, and a model catalog file with provider, upstream model name, per-MTok pricing (fresh/cached/output/cache-write), and prompt-cache TTL. Provider credentials are supplied via env vars like FOREMAN_OPENAI_COMPAT_PROVIDERS, FOREMAN_PROVIDER_<ID>_BASE_URL/_API_KEY, FOREMAN_ANTHROPIC_API_KEY, or FOREMAN_BEDROCK_REGION plus AWS credentials for Claude via Bedrock.
Output is standard OpenAI/Anthropic-compatible API responses (so any existing coding-agent client works unmodified) plus routing metadata in response headers, a queryable ledger (SQLite by default, Postgres for production), and CLI reporting: foreman spend (totals, cache read rate, per-team breakdown), foreman models (model/provider mix), foreman trace <id> (one decision, fully explained), and foreman top (a live terminal view). Every CLI view is backed by GET /v1/foreman/* JSON APIs so custom dashboards can read the same numbers, and --json works on any subcommand.
Integrations
Foreman fronts OpenAI and Anthropic-format client traffic and routes out to any number of providers, translating the tool protocol across formats where needed (e.g. Anthropic-format clients against OpenAI-compatible providers like Qwen or GLM). It documents setup for Claude Code, Codex, OpenCode, and Pi as client harnesses. Provider connectivity covers native Anthropic, OpenAI-compatible providers (configurable per-provider base URL and key), and Claude via AWS Bedrock. Logs are structured JSON carrying request/trace IDs, route metadata, and team attribution, with API keys and prompt payloads explicitly never logged.
Who it's for
Teams and individual developers running AI coding agents against paid LLM APIs who want centralized, policy-driven cost control and a real spend ledger without changing their agent's code - especially teams that want frontier-model quality reserved for planning and high-risk work while routine execution runs on cheaper models, with per-team cost attribution and full routing-decision transparency. It is licensed under Apache-2.0.
Source README
Millwright
The self-hosted LLM router for policy, cache affinity, and spend control in one Rust Binary.
What is Millwright
Millwright is an open-source, self hosted LLM router that runs between your AI applications and the model providers you choose. It accepts OpenAI Chat Completions and Anthropic Messages, then routes each request to an OpenAI-compatible API, Anthropic, or Amazon Bedrock.
You define which models belong in the cheap, mid, and frontier roles. Millwright selects the lowest estimated-cost healthy route the resolved role permits and, when given a session ID, preserves provider/model affinity for prompt-cache reuse.
Millwright is a router, not an agent orchestrator. It does not spawn agents, schedule work, inspect prompts, or rewrite context.
Why Millwright
LLM costs rise quickly when every request reaches the most expensive model and provider changes break prompt-cache reuse. Millwright makes the lower-cost path explicit:
- Control model spend - reserve frontier models for the work your policy says needs them.
- Protect prompt caches - role-scoped affinity keeps each session on a confirmed provider and model while its cache window is warm.
- Keep concurrent agents moving - one session can maintain independent
cheap,mid, andfrontierlanes without serializing requests. - Change providers without changing application code - route compatible traffic across providers behind one endpoint.
- Keep routing inspectable - response headers identify the chosen route; traces and ledger entries retain the evidence when persistence is available.
- Run the data plane yourself - provider credentials stay on your infrastructure, with no required hosted control plane.
Get Started
Quick Start
Rust 1.97+ is required when building from source.
git clone https://github.com/Northwood-Systems/millwright.git
cd millwright
cargo install --path . --locked
millwright init
millwright serve
millwright init walks through provider endpoints, model roles, pricing, and separate workload and operator keys. It stores references to provider-secret environment variables, never the provider credentials themselves.
Connect Claude Code, Codex, OpenCode, Pi, or another compatible client. Choose millwright-fast, millwright-code, or millwright-plan to request the cheap, mid, or frontier role.
Optional Local Demo
The included Docker Compose setup starts Millwright with a mock provider and local demo key:
make compose-up
From another terminal:
curl http://localhost:8080/v1/chat/completions \
-H 'Authorization: Bearer sk-millwright-local-demo' \
-H 'Content-Type: application/json' \
-d '{"model":"millwright-fast","messages":[{"role":"user","content":"Hello from Millwright"}]}'
Architecture · Router API · Policy & Catalog · Deployment · Cost Analysis · Security
Features
Policy-controlled routing
Classify requests with explicit task and risk headers, the client's model selection, or documented defaults. Millwright resolves a cheap, mid, or frontier role, then selects the lowest estimated-cost healthy model that policy allows. High-risk and planning requests always use the frontier role.
Every decision includes the chosen model, provider, route reason, and rejected alternatives. Message content is never inspected to make the decision.
Cache-aware concurrency
Naive per-request routing can erase provider prompt-cache savings by switching models mid-session. Millwright keeps independent cheap, mid, and frontier affinity lanes under one opaque session ID, each with its own cache TTL.
Traffic resolved to different roles no longer fights over one warm route. Same-role calls converge on the last confirmed provider/model, while all requests remain concurrent-Millwright does not serialize agent traffic or orchestrate agent lifecycles.
See where the spend goes
Completed requests normally write usage evidence, cost provenance, provider attempts, and routing evidence to the ledger. SQLite is the local default; PostgreSQL is available for production deployments.
| Command | What it shows |
|---|---|
millwright spend |
Total spend, cache read rate, and per-team breakdown |
millwright models |
Model and provider mix |
millwright trace <id> |
One routing decision, including rejected alternatives |
millwright top |
Live spend and model activity in the terminal |
The same data is available through the authenticated /v1/millwright/* management APIs, with JSON output from spend, models, and trace for dashboards and automation.
Analyze cost opportunities
millwright analyze reads a live Millwright ledger or local Millwright, OpenAI-compatible, or Anthropic JSONL. It produces a self-contained HTML or Markdown report plus schema-versioned JSON.
millwright analyze \
--url http://127.0.0.1:8080 \
--key "$MILLWRIGHT_OPERATOR_KEY" \
--since 30d \
--report analysis.html \
--json-out analysis.json
Reports separate measured cost evidence from modeled candidate economics. Add an explicit catalog and candidate allowlist to compare full-swap costs and a price-floor economic scenario. Candidate results are cost hypotheses-not quality results or production routing recommendations. See Cost Analysis.
Provider and protocol support
OpenAI Chat Completions and Anthropic Messages come in; OpenAI-compatible APIs, native Anthropic, and Amazon Bedrock go out. Same-protocol routes preserve fields Millwright does not model. Cross-protocol routes translate the supported text-and-tools subset and reject unsupported shapes instead of silently dropping data.
Retryable transport failures, timeouts, HTTP 408/429, and HTTP 5xx responses can trigger one bounded failover attempt. Circuit breakers stop unhealthy routes from receiving every request.
Built for self-hosting
Millwright ships as one Rust binary with Docker and Docker Compose support. Provider credentials remain in environment variables, while separate workload and operator keys control inference and management access. Structured logs omit API keys and prompt payloads; request bodies, concurrency, timeouts, and upstream buffers are bounded.
Enterprise
Need help deploying Millwright inside your infrastructure, integrating a provider, or designing a routing policy for your workload?
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.