HALO

✨ RLM-based agent optimizer using production traces✨

Quickstart • What is this? • Benchmarks • Development • Contributing

Quickstart

Install the HALO desktop app with:

curl -fsSL https://inference.net/halo/install.sh | sh

The installer downloads the latest release for your platform and sets up the desktop app. macOS uses a signed, notarized DMG. You can also install directly from the GitHub releases page.

For a full walkthrough of the desktop app, from loading demo traces to running HALO and shipping fixes with a coding agent, see the HALO Desktop guide.

If you're looking for a hosted, plug-and-play version of HALO, please sign up for inference.net and follow the instructions here.

What is this?

HALO is a methodology for building recursively self-improving agent harnesses using RLMs. This repository contains:

• The HALO Desktop App for running HALO locally on your machine.
• Information on HALO methodology.
• A Python package that implements the core HALO-RLM engine. View on PyPI
• A demo project that shows how to build HALO loops for your agents using the Python package. View demo
• Benchmarking examples applying HALO to popular agent benchmarks. (View AppWorld).

HALO Loop

The core HALO loop is surprisingly simple:

1. Collect execution traces from your agent harness. HALO uses OpenTelemetry-compatible tracing.
2. Feed traces into HALO-RLM engine.
3. The engine decomposes the traces to understand common failure modes across harness executions and produces a report with its findings.
4. This report is fed into a coding agent like Cursor or Claude Code to generate and apply a set of changes to your harness.
5. The harness is then re-deployed, more traces are gathered, and the cycle repeats.

HALO is great at finding issues in production agent deployments. We find high-traffic environments tend to generate more data with higher variance across executions, creating the type of issues that HALO is great at identifying.

Why an RLM?

A general-purpose harness like Claude Code is the wrong tool for trace analysis. This isn’t because the model isn’t smart, but because traces can get extremely long, and you need a specialized toolkit in order to make observations about systemic agentic behavior. We noticed in our testing that harnesses like CC would often overfit to an error present in a single/few traces rather than generalize to harness-level problems. This led us to creating a specialized form of a RLM.

Get Started

Install

Install the HALO engine + CLI from PyPI:

pip install halo-engine

# Verify installation
halo --help

Usage

1. Integrate Tracing
2. Collect traces by running your agent
3. Run the HALO engine

export OPENAI_API_KEY=...
# Optional: point HALO at another OpenAI-compatible provider.
export OPENAI_BASE_URL=https://openrouter.ai/api/v1

halo path_to_your_traces.jsonl -p "Diagnose errors you find and suggest fixes"

HALO uses the canonical OpenAI env vars: OPENAI_API_KEY for credentials and OPENAI_BASE_URL for OpenAI-compatible providers. If OPENAI_BASE_URL is unset, HALO uses https://api.openai.com/v1. Run halo --help to see all CLI options. The CLI mirrors the model/provider settings exposed by the Python SDK's
ModelConfig and
ModelProviderConfig.

CLI options

For example:

halo path_to_your_traces.jsonl \
  -p "Diagnose errors you find and suggest fixes" \
  --base-url https://openrouter.ai/api/v1 \
  -H "HTTP-Referer: https://example.com"

Telemetry

HALO can emit OpenInference-shaped traces of its own LLM, tool, and agent activity. It is off by default; nothing is emitted unless you pass --telemetry.

halo TRACE_PATH --prompt "..." --telemetry

When telemetry is enabled, setting INFERENCE_API_KEY uploads spans to inference.net over OTLP. If it is not set, spans are written to a local JSONL file at ./halo-telemetry-{run_id}.jsonl in the current working directory.

We have provided a simple demo and an AppWorld demo.

Python API

The engine exposes four entry points from engine.main. Use whichever
matches the trade-off you want between observability and code
simplicity. The yielded types (AgentOutputItem
and AgentTextDelta) are defined in
engine/models/engine_output.py:

from engine.main import stream_engine_output_async

async for item in stream_engine_output_async(messages, cfg, trace_path):
    logger.info("step", extra={"sequence": item.sequence, "agent": item.agent_name})
    # item.item is an AgentMessage (assistant / tool / etc.)

Benchmarks

HALO is consistently capable of driving improvements on benchmarks, solely by optimizing the harness.

AppWorld

We applied HALO to the AppWorld benchmark, a set of agentic tasks that assess the LLM’s ability to use multi-app services like Spotify, Venmo, file systems, and phone contacts. We tested HALO’s ability to improve harnesses for both Gemini 3 Flash and Sonnet 4.6. We iterated on the harness using the dev split, and then used the test_normal split as a proxy to verify that improvements did not come from overfitting.

The feedback from HALO Engine surfaced failures in the harnesses such as hallucinated tool calls, redundant arguments in tools, refusal loops, and semantic correctness issues. Each issue mapped cleanly to a direct prompt edit. HALO’s claims were independently verified from the source trace files with the findings holding up under scrutiny.

The peak improvements over baseline were substantial for both models. For Gemini 3 Flash, dev SGC went from 36.8% to 52.6% (+15.8 points) and test_normal SGC went from 37.5% to 48.2% (+10.7 points). For Sonnet 4.6, dev SGC went from 73.7% to 89.5% (+15.8 points) and test_normal SGC went from 62.5% to 73.2% (+10.7 points).

Development

Local development against this repo uses uv for dependency management and go-task as the task runner.

Setup

git clone https://github.com/context-labs/HALO
cd HALO
task env:setup

task env:setup installs uv (if missing), syncs the venv from uv.lock, and configures the repo's git hooks. After that, the halo CLI is available via uv run halo ... (or activate .venv/).

Common tasks

Run task --list for the full list. The ones you'll use most: