Catch config drift in your AI coding agent setup

config-drift-checker

CI for your Claude Code setup. Your CLAUDE.md, skills and hooks are how your code gets
written now, and everything underneath them moves without asking: Claude Code ships ~25 releases a
month, and the model behind sonnet can change server-side with no changelog
(it already has, silently, for weeks).
This runs your eval suite on every PR and every release, diffs it against a pinned baseline, and
tells you the moment something stops working: when, why, and what moved.

Proof it works: we broke our own setup, and the repair skill fixed it.
A skill's trigger description rewritten the way a careless PR would: the suite fell 1.00 → 0.56,
the tripwire case read 0.00, and the report named the cause itself: the skill was discovered but
never invoked, so fix the trigger wording, not the packaging. Then the repair skill restored the
behaviour on its own and proved it with a green re-run, for $0.28. Both artifacts are unedited.

See it live

Quick start

One command in the repo whose setup you want protected:

claude plugin marketplace add jameskomo/config-drift-checker && claude plugin install config-drift-checker@jameskomo && claude "/config-drift-checker:setup"

Five minutes: it finds your CLAUDE.md, skills and hooks, writes starter eval cases from them,
smoke-runs them, and writes .cdc.yml plus the GitHub workflow. Prefer never opening Claude at
all? node <plugin-root>/tools/cdc-bootstrap.mjs . runs the same setup headlessly, and
--no-agent scaffolds everything for $0 (blank starter case included) so you fill in the prompts
yourself. Add one secret
(CLAUDE_CODE_OAUTH_TOKEN from claude setup-token to run free on a Pro/Max subscription, or
ANTHROPIC_API_KEY), push, done.

Already have a suite in the claude plugin eval format? One step:

- uses: jameskomo/config-drift-checker/action@v0
  with: { plugin-dir: . }

What you get

• A pinned baseline and a canary. Pinned model + Claude Code version is what every PR is
  diffed against. The canary runs the alias your developers actually get, on the latest Claude
  Code, only when a release actually shipped. It catches what changes underneath you before it
  reaches everyone.
• A diff that doesn't cry wolf. Each case has a noise band learned from its own history: a dip
  inside the band warns instead of failing the build, and guards make sure a real break can never
  hide in the band. Model refusals are labelled as refusals, not setup drift.
• Red cases that diagnose themselves. Every run snapshots which skills the agent could see, so
  a failed skill case says which repair it needs: discovered but never invoked (fix the trigger
  wording) or not discovered (fix the packaging). Watch it happen on a real break.
• Reports that show their work. Every report lists the whole suite including cases a filter or
  budget skipped, every discovered skill and whether it fired, and exactly which checks ran beyond
  a bare claude plugin eval run. And trace-keeper preserves the official runner's transcripts,
  which it otherwise deletes on exit.
• Red check, PR comment, Slack alert, HTML report, drift index. Every grader's verdict with
  its reason, tool calls, cost and turns per case, and a Pages-served index of every case across
  every version.
• Bump PRs. Two green canaries on a new model or version open a PR that moves your pins, with
  the runs attached as evidence. Renovate did this for packages; nobody did it for models.
• Hard budget caps. .cdc.yml caps spend per run and per month; the Action refuses to start
  past the cap. A busy release week cannot drain your key. On a Pro/Max subscription token, runs
  cost $0 API credit.
• Ablation, coverage, repair. With/without-plugin deltas show what each skill is worth;
  coverage shows which of your rules have no test (coverage-min gates it); on a red run a skill
  proposes the smallest setup fix and proves it by re-running the failing cases.
• Fleet and org rollout, serverless. One reusable workflow in your org's .github repo plus a
  three-line caller installs the check on every repo; fleet.mjs folds all their published
  results into one dashboard with a pin policy, and skew from the policy is flagged per repo.

How this relates to claude plugin eval

Claude Code ships an eval runner, and it's good: it runs your cases, grades them, generates
starter cases with init, and writes a report. We build on it, not beside it: cases are in that
exact format, the Action prefers the official runner (bundled fallback for older versions), and a
claude plugin eval . --json out.json result feeds our diff, report and drift index directly.

One sentence: their command answers "does my plugin work right now on my machine"; this answers
"did anything stop working since the baseline, across every release, without me watching".

Trust

Zero npm dependencies (Node builtins only), 88 tests that run against a fake claude with no API
key, every third-party action pinned to a verified commit SHA, CodeQL on every push. Runs on your
runner with your key; nothing is sent to us, because there is no us to send it to. Details in
docs/security.md.

What's here

config-drift-checker/   the plugin: skills (setup · run · write-case · repair) + the tools
  tools/                shim runner · diff · classify · report · dashboard · coverage · watch · gate · promote · trace-keeper · fleet · bootstrap
  test/                 node --test suite, fake claude, npm test
action/                 composite GitHub Action: gate → run → diff → store → PR → repair → alert
examples/komo-stack/    a full example suite with .cdc.yml and baseline results
docs/                   user guide · architecture · eval format & runner · runbook · security

Documentation

Start with the user guide (.cdc.yml reference here); full index in docs/.

Skills in this collection

• repair - SKILL.md
• run: Run the agent-config eval suite locally and compare with the stored baseline; explain why a case went red. Use when the user says "run the evals", "run config-drift-checker", "did the Claude Code u... - SKILL.md
• setup: Set up regression testing (CI) for this repository's Claude Code configuration (plugin, skills, hooks, CLAUDE.md). Use when the user says "set up config-drift-checker", "add evals for my plugin/ski... - SKILL.md
• write-case: Author or fix an eval case (prompt.md, graders/*.md, case.yaml) for a Claude Code plugin, skill, or hook in the official claude plugin eval format. Use when the user says "write an eval for", "ad... - SKILL.md
• spring-boot-conventions: House conventions for writing Spring Boot (Java) backend code - controllers, services, DTOs, pagination, error responses. Use whenever the user asks to write, add, refactor, or review a Spring Boot... - SKILL.md