Diagnose car faults from audio recordings with calibrated ML

cardiag

CLAP, step by step · Isolating the Engine Audio · Model Card · Defense

cardiag is an end-to-end audio-ML pipeline. It scrapes fault-sound clips from YouTube/TikTok, cleans the audio (isolating the mechanical sound from speech, music, and noise), embeds it with a frozen CLAP model, and trains small linear heads to triage the fault. It is exposed as a CLI and a live web app.

https://github.com/user-attachments/assets/23dc48e7-5571-4890-9a6c-4552d907dd8a

This is a proof of concept, and honest about what that means. Diagnosing a car fault from a phone recording is genuinely hard, so cardiag is built as a calibrated triage aid rather than a diagnoser: it tells you whether something sounds wrong, roughly where in the car it is, and a ranked shortlist of likely parts. When the audio won't support a call, it says "uncertain" instead of bluffing.

The real contribution is the cleaning + honest-training recipe, which is reusable on other audio datasets. The modest accuracy here reflects how hard the problem is from crude phone audio (we hit the literature ceiling); the same method reaches 0.93 AUROC on clean engine audio. See docs/DEFENSE.md.

Interactive demos

Two pages visualize the first two stages of the pipeline:

• Isolating the Engine Audio - an interactive look at the clean() cascade pulling a short mechanical span out of noisy YouTube audio (speech, music, road noise).
• CLAP, step by step - how the frozen CLAP model turns those spans into the 512-d embedding the linear heads classify.

What it actually achieves

Measured out-of-sample, leakage-safe (by-video grouped CV over 1,031 video groups; permutation p = 0.0005). These are honest numbers, not a leaderboard.

Full details, and the one head we demoted for failing out-of-sample (knock), are in docs/MODEL_CARD.md.

Quickstart: clone to inference

A fresh clone is immediately usable. A small pre-trained model ships in models/, and a synthetic demo clip is bundled, so nothing needs to be downloaded or scraped.

git clone <this-repo> && cd car-diagnosis
uv venv && source .venv/bin/activate
uv pip install -e ".[scrape,web,dev,viz]"     # Python 3.11

cardiag doctor                 # preflight: what's installed
cardiag train --fixtures       # a working model offline in ~2s (no scrape, no 2 GB download)
cardiag diagnose <clip.wav>    # verdict + where-in-the-car + ranked parts
cardiag serve --model models   # live web app: drop a clip / paste a link, "explain why"

Verify the whole thing end-to-end in an isolated worktree: bash scripts/clone_verify.sh.

How it works

audio ──► clean() cascade ──► CLAP embedding ──► linear heads ──► Diagnosis
          (isolate spans)     (frozen, 512-d)    (fault/region/    (calibrated,
                                                  part/knock)       UNCERTAIN-aware)

There is one segmentation path. Scraped clips, your own recordings (cardiag ingest, any length), and uploads at inference all flow through the same clean() cascade that isolates short mechanical spans. Spans over ~10 s are split into windows so CLAP never silently truncates them. Training and serving share one embedding contract, so there is no train/serve skew.

Usage

cardiag diagnose clip.wav            # full model: verdict + region + ranked parts
cardiag triage   clip.wav            # calibrated engine-vs-running-gear
cardiag clean    clip.wav            # isolate the mechanical sound (no model needed)
cardiag inspect  clip.wav -o r.html  # SEE/HEAR the pipeline: spans, spectrograms, scores
cardiag ingest   ./my_audio --kind fault --cause wheel_bearing   # bring your own audio
cardiag scrape   youtube|tiktok      # build a corpus (Reddit is deprecated — too noisy)
cardiag train                        # train on your corpus

Add --json to any inference command for machine-readable output.

Documentation

• docs/DEFENSE.md - the honest case that a deliberately crude method earns a real triage result.
• docs/MODEL_CARD.md - per-head metrics, intended use, limitations.
• docs/architecture.md - pipeline diagrams.
• docs/scraping-guide.md - start-to-finish corpus building.

Scope & honesty

Valid for social-style / targeted-upload audio (YouTube, TikTok, or a phone clip a user records deliberately). It is not a safety-critical or standalone diagnostic. It is a triage assistant that narrows where to look and is honest about its uncertainty. Model files are joblib artifacts: load only ones you trust.

License: see LICENSE.

scripts/clone_verify.sh

#!/usr/bin/env bash
# Clone-and-run proof: a clueless developer clones with NOTHING and gets a working
# diagnosis + live web app, step by step, in a clean git worktree. Every step is
# the literal command a cloner would run. Exits non-zero on the first failure.
#
#   bash scripts/clone_verify.sh
#
# Needs: git, uv, ffmpeg, yt-dlp on PATH. The ~2GB CLAP weights are reused from the
# shared HuggingFace cache (no re-download). No scrape/train-from-scratch is run.
set -uo pipefail
ROOT="$(cd "$(dirname "$0")/.." && git rev-parse --show-toplevel)"
WT="${TMPDIR:-/tmp}/cardiag_clone_verify"
PASS=0; FAIL=0
step(){ printf "\n\033[1m▶ %s\033[0m\n" "$1"; }
ok(){   printf "  \033[32m✓ %s\033[0m\n" "$1"; PASS=$((PASS+1)); }
bad(){  printf "  \033[31m✗ %s\033[0m\n" "$1"; FAIL=$((FAIL+1)); }
run(){  if eval "$1" >/tmp/cv.out 2>&1; then ok "$2"; else bad "$2 — see below"; tail -8 /tmp/cv.out|sed 's/^/    /'; fi; }

step "0. fresh worktree at HEAD (a clone with nothing)"
cd "$ROOT"
git worktree remove --force "$WT" 2>/dev/null || true
git worktree add --detach "$WT" HEAD >/tmp/cv.out 2>&1 && ok "worktree at $WT" || { bad "worktree add"; exit 1; }
cd "$WT"
# a fresh clone has no scraped data and no user-trained model
rm -rf data/training 2>/dev/null || true

step "1. install (uv venv + editable install with extras)"
uv venv --python 3.11 .venv >/tmp/cv.out 2>&1 && ok "venv" || bad "venv"
source .venv/bin/activate
run "uv pip install -e '.[scrape,web,dev,viz]'" "pip install -e .[scrape,web,dev,viz]"

step "2. quality gates (what a contributor runs)"
run "ruff check src/cardiag tests scripts --output-format=concise" "ruff"
run "mypy" "mypy"
run "python -m pytest -p no:cacheprovider -q" "pytest (offline suite)"

step "3. wheel builds with the bundled model-less data + demo clip"
run "uv build --wheel" "wheel build"
run "python - <<'PY'
import zipfile,glob
z=zipfile.ZipFile(sorted(glob.glob('dist/*.whl'))[-1])
n=z.namelist()
assert any(p.endswith('_fixtures/demo.wav') for p in n), 'demo.wav missing from wheel'
assert any(p.endswith('web/static/index.html') for p in n), 'web UI missing from wheel'
print('wheel contains demo.wav + web UI')
PY" "wheel packages demo clip + web UI"

step "4. onboarding: cardiag doctor + start (offline)"
run "cardiag doctor" "cardiag doctor"
run "cardiag train --fixtures" "cardiag train --fixtures (offline model in ~2s)"

step "5. diagnose the bundled demo clip (NO scrape, NO model needed beyond fixtures/shipped)"
run "cardiag diagnose \"\$(python -c 'from cardiag import paths;print(paths.DEMO_CLIP)')\"" "cardiag diagnose <demo clip>"
run "cardiag clean \"\$(python -c 'from cardiag import paths;print(paths.DEMO_CLIP)')\" --no-music-gate" "cardiag clean <demo clip>"

step "6. the shipped pre-trained model works via the fallback"
run "rm -rf data/training; python -c 'from cardiag import Classifier; Classifier.load(); print(\"shipped model loads\")'" "shipped model auto-fallback (no data/training)"

step "7. live web app: serve + every endpoint"
PORT=8791
cardiag serve --model "$ROOT/models" --port $PORT >/tmp/cv_srv.log 2>&1 &
SRV=$!
for i in $(seq 1 40); do curl -sf http://127.0.0.1:$PORT/health >/dev/null && break; sleep 0.5; done
run "curl -sf http://127.0.0.1:$PORT/health | grep -q '\"model_loaded\":true'" "GET /health (model loaded)"
run "curl -sf http://127.0.0.1:$PORT/favicon.svg | grep -q svg" "GET /favicon.svg"
DEMO="$(python -c 'from cardiag import paths;print(paths.DEMO_CLIP)')"
run "curl -sf -X POST http://127.0.0.1:$PORT/api/diagnose/stream -F file=@$DEMO | grep -q 'event: diagnosis'" "POST /api/diagnose/stream (upload -> diagnosis)"
run "test \$(curl -s -o /dev/null -w '%{http_code}' -X POST http://127.0.0.1:$PORT/api/diagnose/stream --data 'url=http://169.254.169.254/') = 200 && curl -s -X POST http://127.0.0.1:$PORT/api/diagnose/stream --data 'url=http://169.254.169.254/' | grep -q 'only YouTube'" "SSRF guard rejects internal URL"
run "curl -sf -X POST http://127.0.0.1:$PORT/api/explain -F file=@$DEMO | grep -q '\"available\"'" "POST /api/explain (occlusion saliency)"
run "test \$(curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$PORT/api/audio/zzzz) = 400" "GET /api/audio bad-id rejected"
kill $SRV 2>/dev/null

step "summary"
printf "\n  \033[32m%d passed\033[0m, \033[31m%d failed\033[0m\n" "$PASS" "$FAIL"
cd "$ROOT"; git worktree remove --force "$WT" >/dev/null 2>&1 || true
[ "$FAIL" -eq 0 ] && { printf "\n\033[1;32mCLONE-AND-RUN VERIFIED — a stupid developer can do this.\033[0m\n"; exit 0; } || exit 1