Bifrost AI Gateway

Bifrost AI Gateway is a skill for configuring the open-source, Apache-2.0 Bifrost gateway. Written in Go, the gateway exposes more than 20 model providers through an OpenAI-compatible API. It moves provider selection, failover and access governance into a service that applications can address through their existing SDKs.

The workflow covers starting the gateway, adding provider credentials, sending requests, setting fallbacks and load balancing, governing access, enabling caching, observing traffic and connecting MCP servers. Models use the provider/model naming form. Ordered fallback models are tried after the primary fails, each with its own retry budget. Provider network_config controls max_retries and retry backoff; retries default to zero. Credentials, quota or model-access failures advance to another key, while rotation away from a rate-limited key requires retries above zero and multiple keys. Key weights drive weighted random selection; provider weights belong in a virtual key's provider_configs.

Virtual keys define provider and model allowlists, budgets and request or token rate limits. They remain optional until enforce_auth_on_inference is enabled. A key without provider_configs denies all providers. Clients can supply keys through x-bf-vk, Authorization: Bearer, x-api-key or x-goog-api-key. Team and customer budgets can sit above individual keys. Exceeded request rates return 429; exhausted budgets return 402. An inactive or expired key, disallowed provider or model, or blocked MCP tool can produce 403.

Semantic caching requires the semantic_cache plugin and a vector store: Redis/Valkey, Weaviate, Qdrant or Pinecone. Requests need x-bf-cache-key unless default_cache_key is configured. Semantic cache misses add an embedding call; hash mode supports exact matches without embeddings. Request logs show provider, model, tokens, cost and latency. Prometheus metrics are available at /metrics, and the otel plugin exports GenAI-convention spans to an OTLP collector.

Use the skill when an application needs several providers behind one endpoint, configurable failover, per-team spending controls, repeated-prompt caching or consolidated LLM traffic visibility. It also covers exposing several MCP servers through one governed endpoint and routing coding tools through a locally operated gateway.

Bifrost requires someone to run, upgrade and secure the service and its storage. It is unsuitable when running a service is impossible, and it is not a model-training or fine-tuning tool. Provider accounts and API keys remain necessary. Clustering, adaptive routing, advanced guardrails and SSO are Enterprise-only features outside this skill's coverage.

For multiple open-source nodes, provider configuration, budgets and usage remain in each node's memory without synchronization. The guidance uses a shared config.json with config_store disabled; multi-node operation with a shared PostgreSQL store is unsupported. Real-time state synchronization requires Enterprise. Check behavior against documentation for the deployed version because defaults and features change.

Start from provider credentials, intended models, fallback order, key weights, budget and rate-limit requirements, plus any cache, monitoring and MCP connection settings. Provider configuration can be entered through the web UI, management API or config.json. Secret values should be environment references such as env.VAR_NAME rather than literals. A provider key with models set to ["*"] permits all models; an empty model list denies all.

The result is gateway configuration and client routing for inference requests, with optional virtual-key restrictions and cache or telemetry settings. For production use, the skill calls for version pinning, password protection, TLS, enforced virtual keys, per-key controls, configured fallbacks, monitoring and persistent configuration and logs. A fresh instance has neither a dashboard password nor mandatory virtual keys, so access must be secured before exposure beyond localhost.

Provider examples include OpenAI, Anthropic, AWS Bedrock, Google Vertex and Gemini, Azure OpenAI, Mistral, Groq, Ollama and vLLM. OpenAI-compatible clients use the /v1 base route; SDK-specific routes include /openai, /anthropic, /genai and /langchain. Applications change the base URL while retaining their client SDK.

The MCP gateway connects through STDIO, HTTP or SSE and re-exposes tools at /mcp. Virtual-key permissions govern which tools a client sees. Chat-completion tool calls return to the caller and require an explicit POST to /v1/mcp/tool/execute unless Agent Mode auto-execution is configured. Keep explicit execution unless automatic execution of particular tools has been approved.

Claude Code uses the Anthropic-compatible route. Codex CLI uses a Responses API provider with a trailing /v1 in its base URL, responses wire format and the virtual key supplied through OPENAI_API_KEY; the configuration disables required OpenAI account authentication and WebSockets. The skill also calls for logging out of an existing OpenAI account session before using that setup.

The supplied local launch command is:

npx -y @maximhq/bifrost

The gateway and web UI listen on port 8080. Docker is an alternative with configuration and logs persisted in a mounted data directory. Pin a version beyond local evaluation. Set BIFROST_SETUP_TOKEN and enable dashboard password protection before network exposure. Configuration defaults to ~/.config/bifrost for npx unless app-dir is changed, or the mounted data folder for Docker. Only run the local service and management commands in an environment you control.

The skill addresses teams operating applications or coding tools that need shared model-provider routing and governed access. It supports application, team and customer boundaries while leaving responsibility for provider credentials, deployment security and gateway operation with the operator.