Tool

Investigate production incidents and propose root-cause fixes

ninoxAI is an open-source, read-only AI SRE that turns alert storms into incidents with human-approved fix proposals.

Works with checkmkprometheusicinga2zabbixdocker

90
Spark score
out of 100
Updated last month
Version 1.0.0

Add to Favorites

Why it matters

Nightwatch transforms alert storms into actionable incidents by automatically investigating root causes across your live infrastructure and proposing human-approved fixes-all without ever executing changes in production. It serves as a read-only AI SRE layer that correlates alerts from multiple monitoring tools, identifies noisy checks, and uses tool-calling LLMs to gather evidence from Docker, Kubernetes, AWS, Grafana, GitHub, and host systems.

Outcomes

What it gets done

01

Cluster related alerts from multiple monitoring sources into single incidents with cross-tool confirmation

02

Identify flapping, over-sensitive, and never-actioned checks with frequency and acknowledgment analysis

03

Execute read-only investigations across Docker, Kubernetes, AWS, Grafana, and GitHub to build root-cause hypotheses

04

Generate classified fix proposals ranked by risk and blast radius for human approval

Install

Add it to your toolbox

Run in your project directory:

curl -fsSL https://spark.entire.vc/get/ninoxai-nightwatch | bash

Overview

Nightwatch

ninoxAI is an open-source, read-only AI SRE that correlates multi-tool alert noise into incidents, investigates root cause with a tool-calling agent, and proposes human-approved fixes. Use it to correlate alert noise and investigate root cause without giving an agent write access to production. Not an auto-remediation tool - fixes stay human-approved.

What it does

ninoxAI (Nightwatch) is an open-source, local-first, read-only AI SRE layer that sits above monitoring tools - Checkmk, Prometheus, Icinga2, Zabbix, generic webhooks, Docker, Kubernetes, AWS, Grafana, GitHub, Git, and plain VMs - and turns an alert flood into a small number of correlated incidents ("confirmed by N tools" when the same fault fires across multiple tools), flags noisy or flapping checks with evidence, investigates root cause with a tool-calling AI agent reading live systems, and proposes classified, copy-pasteable fixes ranked by risk and blast radius for a human to approve. It is read-only by design: it observes, reasons, and recommends, but never executes a command, acknowledges an alert, changes a threshold, or writes back to production.

When to use - and when NOT to

Use it when alert noise from multiple monitoring tools needs correlating into actual incidents, when noisy or flapping checks need identifying with evidence rather than guesswork, or when a live-system root-cause investigation should run under a typed, read-only capability allowlist rather than giving an agent direct write access. It is explicitly not a remediation or auto-execution tool - gated, governed remediation is on the roadmap, but unconditional auto-execute is not, and every proposed fix is a copyable artifact a human must approve. It can run fully offline with its default "template" LLM provider for summaries and recommendations, but driving the actual investigator agent requires a real tool-calling model - Anthropic is the default for that role, with Mistral or OpenAI/Azure/local models via vLLM, Ollama, or LM Studio also supported.

Capabilities

  • A six-stage pipeline - ingest, normalize, cluster, score noise, recommend, dashboard - with an agentic, read-only root-cause investigator branching off it.
  • The AI SRE investigator runs a ReAct loop (reason, act, observe) over a typed allowlist of read-only capabilities: Docker (containers, logs, stats, inspect), Kubernetes (pods, logs, events, deployments via in-cluster RBAC), AWS (CloudTrail, EC2, security groups, quotas via an IAM read-role), Grafana (PromQL and LogQL over the datasource proxy), GitHub (CI runs, releases, PRs), Git (mirrored-repo commits, diffs, code and history search), and plain-VM hosts (CPU, memory, disk, processes, sockets, log tail).
  • Every investigator action is classified read_only, reversible, or irreversible plus a blast-radius scope - unknown actions coerce to irreversible, never silently treated as safe.
  • Distributed "ninox" runners: thin, outbound-only agents that live inside a remote environment (cluster, VPC, on-prem segment), hold that environment's credentials locally, and dial home to the ninoxAI brain with no inbound firewall hole required - connected runners show up in the Parliament of Owls view.
  • Redaction and secret-scrubbing run before every remote LLM call: hostnames, IPs, UUIDs, emails, and paths become deterministic placeholders, restored only in the final proposed commands, and credentials are one-way scrubbed and never returned.

How to install

cp .env.example .env          # set NINOXAI_SECRET_KEY (one-liner is in the file)
docker compose up --build     # → http://127.0.0.1:8765

A local Python install (pip install -e ".[dev,embeddings]" inside a venv, then python -m ninoxai serve) is also supported for development. Without any live monitoring connected, ninoxai generate-mocks plus ninoxai import/reprocess populates synthetic alert data so the recommendation flow can be tried in under 60 seconds. Beyond the built-in Checkmk, Prometheus Alertmanager, Icinga2, Zabbix, and webhook connectors (PRTG is a stub), new sources can be taught to the AI SRE via any MCP server, a Python capability plugin, or the runner protocol - all routed through the same namespaced, injection-scanned, classification-coerced safety shell.

Who it's for

SRE and platform teams drowning in alert noise across multiple monitoring tools who want AI-assisted root-cause investigation and fix proposals without giving an agent any write access to production - fixes stay human-approved copy-paste artifacts throughout. Fully open source under the Apache License 2.0.

Source README

ninoxAI ninoxAI

ninoxAI

The open-source, read-only AI SRE.
ninoxAI turns alert storms into incidents, investigates root cause over your live systems, and proposes human-approved fixes - without ever touching production.

License: Apache 2.0 Discord Read-only Local-first Python 3.11+

Quickstart · AI SRE · Demo lab · Docs · Discord

ninoxAI turning an alert storm into an incident, root-cause hypothesis, and classified fix proposal


Your monitoring tells you something broke. It pages you at 3am with fifty alerts for one outage and leaves the hard part to you:

What broke, why did it break, and what should we do next?

ninoxAI is a thin, local-first, monitoring-agnostic AI SRE layer that answers that question. It sits above Checkmk, Prometheus, Icinga2, Zabbix, webhooks, Docker, Kubernetes, AWS, Grafana, GitHub, Git and plain VMs, and:

  • 🌊 Turns alert floods into incidents - one incident per outage, "confirmed by N tools", instead of one page per symptom.
  • 🔇 Finds the noisy checks - flapping, over-sensitive, never-actioned - with evidence.
  • 🤖 Investigates root cause - a tool-calling AI agent reads your live systems and forms a root-cause hypothesis.
  • 🧰 Proposes classified fixes - copy-pasteable, ranked by risk and blast radius, for a human to gate.

🔒 Read-only by design

ninoxAI observes, reasons, and recommends - it never executes anything. No commands run, no alerts acked, no thresholds changed, no write-back to production. Every fix is a copyable artifact a human approves. Gated, governed remediation is on the roadmap; unconditional auto-execute is not.


⚡ Quickstart

Try it in 60 seconds - no LLM, no API keys, fully offline:

cp .env.example .env          # set NINOXAI_SECRET_KEY (one-liner is in the file)
docker compose up --build     # → http://127.0.0.1:8765

# No live monitoring? Watch it triage synthetic alert noise:
docker compose exec ninoxai ninoxai generate-mocks
docker compose exec ninoxai ninoxai import data/mock_alerts.json
docker compose exec ninoxai ninoxai reprocess
# → /recommendations now shows reasoned threshold + flapping fixes
Local Python install (for development)
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\Activate.ps1
pip install -e ".[dev,embeddings]"

python -m ninoxai generate-mocks
python -m ninoxai import data/mock_alerts.json
python -m ninoxai reprocess
python -m ninoxai serve            # → http://127.0.0.1:8765

Then light up the AI SRE: point ninoxAI at a tool-calling LLM (Anthropic / OpenAI / Mistral / a local Ollama) and connect your systems - either directly or via a ninox runner (below). The full end-to-end scenario - real monitoring tools and live investigator capabilities (Docker/Kubernetes/host/AWS/Grafana/GitHub) against a genuinely failing workload - lives in lab/.


🔭 How it works

ingest → normalize → cluster → score noise → recommend → dashboard
                                                  ↓
                       agentic, read-only root-cause investigator
Stage What happens
ingest Read-only adapters pull non-OK alerts from each source + JSON/CSV import.
normalize Maps every source onto one schema + message fingerprint.
cluster Groups by host / service / severity / time-window. Semantic embeddings optional.
noise Frequency, ack-rate, ticket-rate, short-recovery, flapping → one 0-1 score.
recommend Rule-based tuning recommendations with rationale + evidence.
investigate A tool-calling LLM gathers live evidence → root-cause hypothesis + classified fixes.

Cross-tool correlation: the same fault fires in every tool. The Incidents view groups clusters that share (host, severity, time-window) into one incident - "confirmed by N tools" - read-only, no merge.


🤖 The AI SRE investigator

ninoxAI's standout capability. A tool-calling LLM drives a typed allowlist of read-only capabilities (a ReAct loop on native function-calling - reason → act → observe), builds a root-cause hypothesis from live evidence, and proposes classified fixes a human approves.

Capability Reads (all read-only)
🐳 Docker containers, logs, stats, inspect
☸️ Kubernetes pods, logs, events, deployments (in-cluster RBAC)
☁️ AWS CloudTrail change events, EC2, security groups, quotas (IAM read-role)
📈 Grafana PromQL + LogQL over the datasource proxy
🐙 GitHub CI runs, releases, PRs - change-event RCA
🌿 Git mirrored repos: commits, diffs, code & history search
🖥️ Host CPU / mem / disk / processes / sockets / log tail (plain VMs)
  • Every action is classified read_only · reversible · irreversible + a scope (blast radius). Unknown coerces to irreversible - never silently auto.
  • Pre-grounded: the agent starts with a compact brief of your environment, so it diagnoses instead of rediscovering.
  • Hardened: untrusted logs/diffs are injection-shielded, secrets are one-way scrubbed, and a grounding gate caps confidence when claims aren't backed by evidence.

Run it live-streaming in the agent console (/agent) or from the CLI. → Investigator internals


🦉 Distributed ninoxes - the agent's eyes, anywhere

The agent can investigate systems it can't reach directly. A ninox is a thin, outbound-only runner that lives inside one environment (cluster, VPC, on-prem segment), holds that environment's credentials locally, and dials home to the brain - no inbound firewall hole. It advertises a read-only capability surface the brain calls as if local.

   ┌────────────────────┐                         ┌────────────────────┐
   │    ninoxAI brain    │  ◀── outbound only ───  │     ninox runner    │
   │  dashboard · API    │     (the ninox dials    │  inside k8s/Docker/ │
   │  incidents · RCA    │      home; no inbound    │  AWS/on-prem/VM     │
   │  AI SRE investigator│      firewall hole)      │  credentials stay   │
   └────────────────────┘  ◀── read-only evidence  │  local              │
                                                     └────────────────────┘

Capabilities self-select by environment - one binary, the right tools for the box it lands on. Connected ninoxes show up in the Parliament of Owls (/parliament). → Deployment & on-prem


🔌 Connectors

All adapters are read-only - no ack, no downtime, no write-back. Configured in the UI (/connections), credentials Fernet-encrypted.

Checkmk Prometheus Alertmanager Icinga2 Zabbix Generic Webhook PRTG
⛔ stub

Want to teach the AI SRE to read your stack (Jira, Sentry, Postgres…)? Point it at any MCP server, write a Python capability plugin, or expose tools via the runner protocol - every external tool runs through the same safety shell (namespaced, injection-scanned, classification-coerced). → Extending capabilities


🧠 LLM providers

Default is template - fully offline: no LLM, no network, no API keys, no tracking. It works out of the box for summaries/recommendations but deliberately can't drive the agent (that needs tool-calling). Pick a remote per role - a cheap model for high-volume summaries, a strong one for the rare investigation:

Provider Notes
template offline - no LLM, no network. Default.
mistral cost-efficient, EU-hosted
anthropic strong tool-calling - default for the investigator
openai OpenAI, Azure, and local LLMs (vLLM / Ollama / LM Studio) via base URL

Redaction + secret-scrubbing run before every remote call - hostnames, IPs, UUIDs, emails, paths become deterministic placeholders, restored only in proposed commands; credentials are one-way scrubbed and never returned. → Technical architecture


🛠️ Development

Full CLI reference, test setup, and lint rules live in docs/development.md.


🤝 Contributing

Every contributor is an Owl. 🦉 Pull requests, connector adapters, capability providers, and bug reports are all welcome - see CONTRIBUTING.md.

Community: Join the parliament on Discord.

📜 License

ninoxAI is fully open source under the Apache License 2.0 - free to use, self-host, fork, and build on, in open or closed projects alike.

The owl observes; the human decides. 🦉

Discussion

Questions & comments · 0

Sign In Sign in to leave a comment.