Accelerate multi-agent GPU inference with zero-copy context sharing

Context-Stitcher 🪡

Zero-Copy Context Bridging Gateway for Multi-Agent GPU Inference.

💡 Core Value Proposition

In multi-agent collaborative workflows, separate agents often process the same long text context sequentially. For example:

1. Agent A (Legal Auditor): Reads a 200-page contract and runs compliance analyses (populating the GPU KV Cache).
2. Agent B (Financial Compliance): Reads the same 200-page contract and audits financial liabilities.

Under standard inference engines, Agent B is forced to repeat the expensive prefill phase, duplicate GPU activations, and suffer from high Time-to-First-Token (TTFT) latency.

Context-Stitcher solves this by bridging caches at the memory level:

• Context Topological Hashing: Segmenting prompts into physical block-sizes and mapping them to cryptographic fingerprints (Merkle-chains).
• Zero-Copy Block Stitching: Bypassing prefill for matched prefixes by mapping the logical attention table of Agent B directly to the physical GPU memory address of Agent A's cache blocks.
• Zero-Trust Secure Gate: Enforcing boundary control lists so unauthorized agent sessions cannot access shared physical blocks.

📊 Performance Profiles

Below is the benchmark analysis of Context-Stitcher compared to standard vLLM cold-prefills when executing consecutive agents over a shared 200-page document:

⚡ TTFT Prefill Latency (Agent B Response Time) — Lower is better
Baseline (vLLM Cold):   ██████████████████████████████  1200 ms
Context-Stitcher:       █  48 ms ( 25.0x Prefill Speedup! 🚀 )

💾 GPU Physical Cache Blocks Allocated (Total VRAM) — Lower is better
Baseline (vLLM Cold):   ██████████████████████████████  53 blocks (No sharing)
Context-Stitcher:       ████████████████  30 blocks ( 43.4% Memory Saved! 📉 )

⚙️ Installation & Quick Start

1. Install Dependencies

pip install -r requirements.txt

2. Launch the Gateway & Dashboard

python run.py

Once booted, the gateway routes are active on http://localhost:8000:

• API Proxy Gateway: http://localhost:8000/v1/chat/completions
• Real-time Developer Console: Open http://localhost:8000 in your web browser.

🖥️ Visual Developer Portal

Context-Stitcher includes a responsive developer portal to monitor physical cache block states (idle, private allocations, shared/stitched pages, security alarms) in real time.

🛠️ Client Integration & Usage Guide

Context-Stitcher supports Python SDK Decorators and OpenAI-Compatible REST APIs for cross-application integrations:

Pattern A: Python SDK Decorator (For Python-based Agent Orchestration)

If your agent pipelines are written in Python, you can utilize the StitcherMesh and @stitch_agent decorators to link context memory:

from context_stitcher import StitcherMesh, stitch_agent

# 1. Initialize the pointer-sharing inference control mesh (ref: vLLM)
mesh = StitcherMesh(backend="vllm", model="meta-llama/Llama-3.1-8B-Instruct")

# Configure secure gate policy: allow Agent B to read Agent A's KV Cache blocks
mesh.sg.add_policy("agent_a", "agent_b")

# 2. Decorate your agent workflows
@stitch_agent(mesh)
def agent_a():
    prompt = "[Long Document Context...]\nAnalyze the intellectual property clauses."
    # First execution: Prefills the cache and registers topological hashes
    res = mesh.generate(prompt=prompt, fingerprint="legal_doc_v1")
    return "legal_doc_v1"

@stitch_agent(mesh)
def agent_b(context_fingerprint):
    prompt = "[Long Document Context...]\nEvaluate the financial compliance risk."
    # Subsequent execution: Automatically stitches blocks, bypassing prefill phase
    res = mesh.generate(prompt=prompt, fingerprint=context_fingerprint)
    print(f"Agent B Response: {res['generated_text']}")
    print(f"Time Saved: {res['prefill_time_saved_ms']}ms")

Pattern B: OpenAI-Compatible REST API (For multi-language clients, Dify, or Flowise)

The gateway exposes standard OpenAI endpoints. Point your LLM client base URL to Context-Stitcher to activate sharing.

1. Python OpenAI SDK Client:

from openai import OpenAI

# Direct client to the Context-Stitcher proxy gateway
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

# Send standard completions request with agent_id metadata inside extra_body
response = client.chat.completions.create(
    model="context-stitcher-sim",
    messages=[
        {"role": "user", "content": "[Long Document Context...]\nEvaluate compliance risk."}
    ],
    extra_body={
        "agent_id": "AgentB",                # Identifies the requesting Agent
        "session_id": "session_legal_audit"  # Identifies the shared session cache
    }
)

print("Generated Output:", response.choices[0].message.content)

2. Raw HTTP cURL Request:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "context-stitcher-sim",
    "messages": [
      {"role": "user", "content": "[Long Document Context...]\nEvaluate compliance risk."}
    ],
    "agent_id": "AgentB",
    "session_id": "session_legal_audit"
  }'

Pattern C: Controlling Security Rules via API (Zero-Trust Gate)

You can inspect, add, or revoke access authorization rules dynamically between agents.

1. Retrieve all active policy rules:

curl -X GET http://localhost:8000/policies

2. Authorize AgentB to stitch AgentA's KV Cache:

curl -X POST http://localhost:8000/policy \
  -H "Content-Type: application/json" \
  -d '{"owner_agent": "AgentA", "allowed_reader": "AgentB"}'

3. Revoke AgentB's permission to read AgentA's Cache:

curl -X DELETE http://localhost:8000/policy \
  -H "Content-Type: application/json" \
  -d '{"owner_agent": "AgentA", "allowed_reader": "AgentB"}'