Build AI-powered SEO content engine with GSC feedback loops

The AI SEO Playbook: SEO + GEO + AEO

The open-source operating system behind a content site that reached 11.08M impressions and 44.8K clicks in 90 days. That's 7.7× the impressions and 12× the clicks of the previous 90 days. Every number here is verifiable in RESULTS.md.

This isn't a list of tips. It's the methodology, the scripts, and the guardrails we actually run: tested, measured against controls, and including the mistakes that cost us traffic so you don't repeat them.

The 2026 reality most SEO advice misses

On our blog, human-shaped search queries convert at 0.95%. Machine-shaped ones (AI research agents, scrapers, SEO tools) convert at 0.01%, a 95× gap, and they make up almost half of the page-one impressions. So:

• Your blended CTR is no longer a title metric. Retitling a page whose impressions come from an AI agent does nothing. human-query-split shows you which pages are really underperforming for humans.
• Those agent impressions still matter. They are AI engines reading you. Winning GEO/AEO (being the source ChatGPT, Perplexity, Claude and Google AI Overviews cite) takes extractable, dated, verified facts. Chapter 1 covers how.
• Most "wins" aren't. Measured against untouched pages, our title rewrites outgrew the control by roughly 19-48 points (depending on the window). Our snippet rewrites did nothing, and our "freshness refreshes" only looked like losses because they picked fading news. Chapter 2 shows how to tell the difference.

Who this is for

Founders, marketers and engineers running content, programmatic or news sites, especially with AI in the content pipeline, who want rankings and AI citations without gambling on unmeasured tactics.

Quick start

Before you start: the Search Console scripts need a Google Cloud project with the Search Console API enabled and access to your property. Follow docs/setup-gsc.md first (about 10 minutes).

git clone https://github.com/TraceCohenTech/ai-seo-playbook && cd ai-seo-playbook
npm ci

# Authenticate with Search Console. Needs a Google Cloud project with the
# Search Console API enabled; see docs/setup-gsc.md for the step-by-step setup.
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/webmasters.readonly,https://www.googleapis.com/auth/cloud-platform

# Is your low CTR a title problem or an AI-agent problem?
node scripts/human-query-split.mjs --site sc-domain:yoursite.com

# Daily drop alarm for cron/CI (exit code 2 = alert)
node scripts/traffic-guard.mjs --site sc-domain:yoursite.com --webhook https://ntfy.sh/your-topic

# Are the URLs in your sitemaps actually indexable?
node scripts/sitemap-health.mjs --sitemap https://yoursite.com/sitemap.xml

Every script prints its full documentation with --help.

What does --dir point at?

Some scripts read your site's source files instead of (or as well as) Search Console. --dir is the folder in your website's own code repo that holds the pages, not a cache and not a special export format:

Run the scripts from this repo and pass the path to your site's repo, e.g. node scripts/thin-content-detector.mjs --dir ../my-site/src/app. Use --ext to change which file types are read. For Markdown, a file's URL is its path inside --dir, so if content/blog/foo.md is served at /blog/foo, add --url-prefix /blog where the script supports it.

Setup details are in docs/setup-gsc.md.

The playbook

Plus a prompt library with sourcing rules built in, and a bot-traffic guide.

The toolkit

Understand your traffic (Search Console)

Measure changes honestly

Protect what you've built

GEO / AEO and data quality

Content quality

Feeds and indexing

To run the weekly report on a schedule, copy examples/workflows/weekly-seo-report.yml into your site's repo.

Configs (quality gates, title rules, refresh rules, schema rules…) live in config/, and JSON-LD templates in schemas/. Sample outputs for the scripts are in samples/, generated from test fixtures.

Principles

1. Measure against a control, or it didn't happen.
2. Separate humans from machines before judging CTR.
3. Every fact needs a source you can re-check. AI answer engines reward being right.
4. Structured data mirrors visible content. No synthetic dates or invisible FAQs.
5. Guardrails before automation. If a machine can publish it, a machine should also be able to catch it.
6. No tricks that need hiding. Nothing here disguises automation or games a policy.

Results

Full numbers, windows and methods are in RESULTS.md. The site behind them is valueaddvc.com, a startup and venture-capital intelligence platform with ~1,100 articles, a daily news feed and thousands of programmatic company and investor pages.

Roadmap and community

The playbook and scripts are and stay free (MIT). See ROADMAP.md for what's next, including hosted audits. Contributions with real numbers and windows are very welcome; see CONTRIBUTING.md.

Built by Trace Cohen at Value Add VC. If this helped you, a ⭐ helps others find it.

scripts/human-query-split.mjs

#!/usr/bin/env node
/**
 * Human Query Split: is your low CTR a title problem or an AI-agent problem?
 *
 * Blended CTR in 2026 is polluted by AI research agents and scrapers that generate
 * impressions but never click. This script separates each page's impressions into
 * human-shaped vs machine-shaped queries (lib/query-classifier.mjs), then flags the pages
 * that underperform FOR HUMANS: they rank on page one for human queries but earn under half
 * the site's own median human CTR at that position. Those are the only pages worth a
 * retitle. Everything else is agent traffic that no title will convert.
 *
 * Usage:
 *   node scripts/human-query-split.mjs --site sc-domain:example.com [--days 28] [--min-human-impr 300] [--output out.json]
 */
import { cli } from '../lib/cli.mjs';
import { writeFileSync } from 'node:fs';
import { gscClient, queryAll, isoDay, daysAgo } from '../lib/gsc.mjs';
import { splitQueries } from '../lib/query-classifier.mjs';

const a = cli(import.meta.url, {
  site: { type: 'string', required: true }, days: { type: 'string', default: '28' },
  'min-human-impr': { type: 'string', default: '300' }, 'path-contains': { type: 'string' }, output: { type: 'string' },
});

const sc = await gscClient();
const endDate = isoDay(daysAgo(3)), startDate = isoDay(daysAgo(3 + Number(a.days)));
const filters = a['path-contains'] ? [{ dimension: 'page', operator: 'contains', expression: a['path-contains'] }] : undefined;
const rows = await queryAll(sc, a.site, { startDate, endDate, dimensions: ['page', 'query'], filters });

const byPage = new Map();
for (const r of rows) {
  const [page, query] = r.keys;
  (byPage.get(page) || byPage.set(page, []).get(page)).push({ query, impressions: r.impressions, clicks: r.clicks, position: r.position });
}

const pages = [];
let totH = 0, totHC = 0, totM = 0, totMC = 0;
for (const [page, qs] of byPage) {
  const { human, machine } = splitQueries(qs);
  const s = (arr, k) => arr.reduce((n, x) => n + x[k], 0);
  const hi = s(human, 'impressions'), hc = s(human, 'clicks'), mi = s(machine, 'impressions'), mc = s(machine, 'clicks');
  totH += hi; totHC += hc; totM += mi; totMC += mc;
  const hpos = hi ? human.reduce((n, x) => n + x.position * x.impressions, 0) / hi : null;
  pages.push({ page, humanImpr: Math.round(hi), humanClicks: hc, humanCtr: hi ? hc / hi : 0, humanPos: hpos,
    machineImpr: Math.round(mi), machineShare: hi + mi ? mi / (hi + mi) : 0,
    topHumanQueries: human.sort((x, y) => y.impressions - x.impressions).slice(0, 3).map((x) => x.query) });
}

// The site's own expected human CTR by position (median of pages with enough human impressions).
const buckets = {};
for (const p of pages) if (p.humanImpr >= 100 && p.humanPos) (buckets[Math.min(10, Math.max(1, Math.round(p.humanPos)))] ||= []).push(p.humanCtr);
const median = (arr) => { const x = [...arr].sort((m, n) => m - n); return x.length ? x[x.length >> 1] : 0; };
const expected = Object.fromEntries(Object.entries(buckets).map(([b, v]) => [b, median(v)]));

const minH = Number(a['min-human-impr']);
const retitle = pages.filter((p) => p.humanImpr >= minH && p.humanPos && p.humanPos <= 8 && p.humanCtr < 0.5 * (expected[Math.round(p.humanPos)] || 0.01))
  .map((p) => ({ ...p, expectedCtr: expected[Math.round(p.humanPos)] || 0 }))
  .sort((x, y) => y.humanImpr * (y.expectedCtr - y.humanCtr) - x.humanImpr * (x.expectedCtr - x.humanCtr));
const agentDominated = pages.filter((p) => p.machineImpr >= 1000 && p.machineShare >= 0.8).sort((x, y) => y.machineImpr - x.machineImpr);

const pct = (x) => (100 * x).toFixed(2) + '%';
console.log(`\nWindow ${startDate}..${endDate} · ${byPage.size} pages · ${rows.length} page×query rows (visible queries only)`);
console.log(`Human-shaped:   ${Math.round(totH).toLocaleString()} impr, CTR ${pct(totHC / Math.max(totH, 1))}`);
console.log(`Machine-shaped: ${Math.round(totM).toLocaleString()} impr, CTR ${pct(totMC / Math.max(totM, 1))}`);
console.log(`\nReal retitle opportunities (human CTR < 50% of site median at that position): ${retitle.length}`);
for (const p of retitle.slice(0, 20)) console.log(`  ${p.page}\n    human ${p.humanImpr} impr · ${pct(p.humanCtr)} vs ${pct(p.expectedCtr)} expected · pos ${p.humanPos.toFixed(1)} · "${p.topHumanQueries.join('", "')}"`);
console.log(`\nAgent-dominated pages (≥80% machine impressions; a new title won't help): ${agentDominated.length}`);
for (const p of agentDominated.slice(0, 10)) console.log(`  ${p.page} · ${p.machineImpr} machine impr (${pct(p.machineShare)})`);

if (a.output) writeFileSync(a.output, JSON.stringify({ window: { startDate, endDate }, totals: { human: { impr: totH, clicks: totHC }, machine: { impr: totM, clicks: totMC } }, expectedHumanCtrByPosition: expected, retitle, agentDominated }, null, 2));

scripts/traffic-guard.mjs

#!/usr/bin/env node
/**
 * Traffic Guard: daily Search Console drop alarm (the one we wish we'd had).
 *
 * A middleware change once served `X-Robots-Tag: noindex` on every public page for 28 hours.
 * Daily impressions fell ~55% before anyone noticed, because 28-day rolling reports smooth a
 * cliff into a slope. This checks the last COMPLETE day (today-3; GSC's newest days are
 * provisional) against three baselines:
 *   1. same weekday, previous 2 weeks (handles weekend seasonality)
 *   2. previous 7 days
 *   3. same weekday 4-5 weeks back, clicks only (after an incident the short baselines
 *      are themselves depressed, which would hide a slow or partial recovery)
 * It alerts when the day is below --threshold (0.75) of BOTH 1 and 2, or clicks are below
 * --long-threshold (0.70) of 3. Exit 0 = ok, 2 = alert (wire it to CI or cron), 1 = error.
 * Optional --webhook posts the alert (ntfy, Slack incoming webhook, …).
 *
 * Usage: node scripts/traffic-guard.mjs --site sc-domain:example.com [--day YYYY-MM-DD] [--webhook https://ntfy.sh/your-topic]
 */
import { cli } from '../lib/cli.mjs';
import { gscClient, queryAll, isoDay, daysAgo } from '../lib/gsc.mjs';

const a = cli(import.meta.url, {
  site: { type: 'string', required: true }, day: { type: 'string' }, threshold: { type: 'string', default: '0.75' },
  'long-threshold': { type: 'string', default: '0.70' }, webhook: { type: 'string' },
});

const D = a.day ? new Date(a.day + 'T00:00:00Z') : daysAgo(3);
const back = (n) => { const x = new Date(D); x.setUTCDate(x.getUTCDate() - n); return isoDay(x); };
const sc = await gscClient();
const rows = await queryAll(sc, a.site, { startDate: back(38), endDate: isoDay(D), dimensions: ['date'] });
const by = new Map(rows.map((r) => [r.keys[0], r]));
const median = (xs) => { const s = xs.filter(Number.isFinite).sort((x, y) => x - y); return s.length ? (s.length % 2 ? s[s.length >> 1] : (s[s.length / 2 - 1] + s[s.length / 2]) / 2) : NaN; };
const T = Number(a.threshold), LT = Number(a['long-threshold']);
const day = by.get(isoDay(D));
const out = [];
let alert = !day;
for (const m of ['clicks', 'impressions']) {
  const v = day?.[m];
  const weekday = median([7, 14].map((n) => by.get(back(n))?.[m]));
  const prior7 = median([1, 2, 3, 4, 5, 6, 7].map((n) => by.get(back(n))?.[m]));
  const long = median([28, 35].map((n) => by.get(back(n))?.[m]));
  const hit = (v < T * weekday && v < T * prior7) || (m === 'clicks' && Number.isFinite(long) && v < LT * long);
  alert ||= hit;
  out.push(`${m}=${v ?? 'MISSING'} (weekday ${weekday} / 7d ${prior7}${m === 'clicks' ? ` / 4-5wk ${long}` : ''})${hit ? ' ⚠' : ''}`);
}
const line = `traffic-guard ${isoDay(D)} ${out.join(' · ')} → ${alert ? 'ALERT' : 'ok'}`;
console.log(line);
if (alert && a.webhook) {
  const slack = a.webhook.includes('hooks.slack.com');
  await fetch(a.webhook, { method: 'POST', headers: slack ? { 'content-type': 'application/json' } : { Title: 'Search traffic drop' }, body: slack ? JSON.stringify({ text: line }) : line }).catch(() => {});
}
process.exit(alert ? 2 : 0);

scripts/sitemap-health.mjs

#!/usr/bin/env node
/**
 * Sitemap Health: are the URLs you submit actually indexable?
 *
 * A sitemap should only list 200, indexable, self-canonical URLs. Listing duplicates,
 * noindexed or redirected pages wastes crawl budget and shows up as sitemap errors (one
 * foreign-canonical story in a news sitemap was the entire "errors=1" we chased). This
 * validates each sitemap's XML, reports duplicates, then SAMPLES N URLs per sitemap and
 * checks status, X-Robots-Tag, meta robots and canonical.
 *
 * It samples and paces on purpose. Crawling thousands of your own URLs at speed can trip
 * your host's bot protection (it happened to us on Vercel), and then the audit itself becomes
 * the incident.
 *
 * Usage: node scripts/sitemap-health.mjs --sitemap https://example.com/sitemap.xml [--sitemap …] [--sample 15] [--delay-ms 1000]
 */
import { cli } from '../lib/cli.mjs';

const a = cli(import.meta.url, {
  sitemap: { type: 'string', multiple: true, required: true }, sample: { type: 'string', default: '15' }, 'delay-ms': { type: 'string', default: '1000' },
});
const UA = 'Mozilla/5.0 (compatible; sitemap-health/1.0; +https://github.com/TraceCohenTech/ai-seo-playbook)';
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
const locs = (xml) => [...xml.matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)].map((m) => m[1].replace(/&amp;/g, '&'));
let problems = 0;

for (const sm of a.sitemap) {
  const res = await fetch(sm, { headers: { 'user-agent': UA } });
  const xml = await res.text();
  const wellFormed = /^\s*<\?xml|<urlset|<sitemapindex/.test(xml) && (xml.match(/<url>/g)?.length ?? 0) === (xml.match(/<\/url>/g)?.length ?? 0);
  let urls = locs(xml);
  if (/<sitemapindex/.test(xml)) { console.log(`${sm}: sitemap index with ${urls.length} child sitemaps (pass the children directly)`); continue; }
  const dupes = urls.length - new Set(urls).size;
  console.log(`\n${sm}: HTTP ${res.status} · ${wellFormed ? 'well-formed' : 'MALFORMED'} · ${urls.length} URLs · ${dupes} duplicates`);
  if (res.status !== 200 || !wellFormed || dupes) problems++;
  urls = [...new Set(urls)].sort(() => Math.random() - 0.5).slice(0, Number(a.sample));
  for (const u of urls) {
    await sleep(Number(a['delay-ms']));
    try {
      const r = await fetch(u, { headers: { 'user-agent': UA }, redirect: 'manual' });
      const html = r.status === 200 ? await r.text() : '';
      const xr = r.headers.get('x-robots-tag') || '';
      const meta = (html.match(/<meta[^>]+name=["']robots["'][^>]+content=["']([^"']+)/i) || [])[1] || '';
      const canon = (html.match(/<link[^>]+rel=["']canonical["'][^>]+href=["']([^"']+)/i) || [])[1] || '';
      const issues = [];
      if (r.status !== 200) issues.push(`status ${r.status}${r.headers.get('location') ? ' → ' + r.headers.get('location') : ''}`);
      if (/noindex/i.test(xr)) issues.push(`X-Robots-Tag: ${xr}`);
      if (/noindex/i.test(meta)) issues.push(`meta robots: ${meta}`);
      if (canon && canon.replace(/\/$/, '') !== u.replace(/\/$/, '')) issues.push(`canonical → ${canon}`);
      if (issues.length) { problems++; console.log(`  ✗ ${u}\n      ${issues.join(' · ')}`); }
    } catch (e) { problems++; console.log(`  ✗ ${u} (${e.message})`); }
  }
  console.log(`  sampled ${urls.length}`);
}
console.log(`\n${problems ? `${problems} problem(s)` : 'all sampled URLs are 200, indexable and self-canonical'}`);
process.exit(problems ? 2 : 0);

scripts/thin-content-detector.mjs

#!/usr/bin/env node

/**
 * Thin Content Detector
 *
 * Finds content files with little visible text, so you can decide whether to expand,
 * merge or noindex them. Thin pages dilute a site's overall quality signals and often
 * end up as "Crawled - currently not indexed" in Search Console. (Crawl budget is rarely
 * the issue: Google says it matters mainly for very large sites.)
 *
 * The word count is of VISIBLE text: imports/exports, frontmatter, code, JSX attributes,
 * JSX expressions, <script>/<style> and HTML tags are removed first. For Next.js
 * App Router pages (page.tsx) only the text inside JSX elements is counted. Text that a
 * page renders from data at runtime (arrays, CMS, props) is not visible in the source and
 * is not counted, so check rendered HTML (save it as .html and scan that) when in doubt.
 * Dynamic route files (a [param] directory, e.g. blog/[slug]/page.tsx) are skipped for that reason.
 *
 * Usage:
 *   node scripts/thin-content-detector.mjs --dir ./content
 *   node scripts/thin-content-detector.mjs --dir ./src/app --min-words 250 --min-age-days 180
 *
 * Options:
 *   --dir <path>            Directory to scan (required).
 *   --ext <list>            Comma-separated extensions (default: .md,.mdx,.tsx,.jsx,.html).
 *                           Dot-directories, node_modules and binary files are always skipped.
 *   --min-words <n>         Pages below this many visible words are "thin" (default: 300).
 *                           Pages below a third of it are "critical".
 *   --min-age-days <n>      A critical page is only labelled NOINDEX_CANDIDATE when it is at
 *                           least this old (default: 90). Age comes from a frontmatter / metadata
 *                           date (date, publishedAt, datePublished, ...) or, failing that, the
 *                           file's mtime (unreliable in fresh git checkouts; reported as ageSource).
 *   --output <file>         JSON report path (default: thin-content.json).
 *
 * Labels (heuristics; nothing here is applied automatically):
 *   NOINDEX_CANDIDATE  critical word count AND older than --min-age-days. This script has no
 *                      traffic data: confirm the page has ~zero clicks/impressions in Search
 *                      Console (or use content-audit.mjs, which joins GSC) before noindexing.
 *   REVIEW             thin, link-heavy with little prose, or too new to judge.
 *
 * Output: a JSON array of flagged files:
 *   { file, wordCount, linkCount, ageDays, ageSource, issues: [{type, detail, severity}], recommendation }
 *
 * Exit codes: 0 success (findings are not an error), 1 error.
 *
 * The file also exports the shared content helpers (walkContentFiles, extractVisibleText,
 * countWords, pageAge, ...) used by the other content scripts.
 */

import { readdirSync, readFileSync, statSync, writeFileSync, realpathSync } from 'node:fs';
import { join, extname, relative, sep } from 'node:path';
import { pathToFileURL } from 'node:url';
import { cli, fail } from '../lib/cli.mjs';

// ───────────────────────── shared content helpers ─────────────────────────

export const DEFAULT_EXTS = ['.md', '.mdx', '.tsx', '.jsx', '.html'];

/** Parse "--ext .md,mdx" into ['.md', '.mdx']. */
export function parseExts(value, fallback = DEFAULT_EXTS) {
  if (!value) return fallback;
  return value.split(',').map((e) => e.trim().toLowerCase()).filter(Boolean).map((e) => (e.startsWith('.') ? e : `.${e}`));
}

/** Recursively list files with one of `exts`, skipping dot-entries and node_modules. */
export function walkContentFiles(dir, exts = DEFAULT_EXTS) {
  const files = [];
  for (const entry of readdirSync(dir).sort()) {
    if (entry.startsWith('.') || entry === 'node_modules') continue;
    const full = join(dir, entry);
    let st;
    try { st = statSync(full); } catch { continue; }
    if (st.isDirectory()) files.push(...walkContentFiles(full, exts));
    else if (st.isFile() && exts.includes(extname(entry).toLowerCase())) files.push(full);
  }
  return files;
}

/** Read a file as UTF-8 text, or return null if it looks binary (contains a NUL byte). */
export function readTextFile(file) {
  const buf = readFileSync(file);
  if (buf.includes(0)) return null;
  return buf.toString('utf8').replace(/^/, '');
}

const ENTITIES = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', rsquo: '’', lsquo: '‘', ldquo: '“', rdquo: '”', mdash: '—', ndash: '–', hellip: '…', copy: '©' };
export function decodeEntities(s) {
  return s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
    if (e[0] === '#') {
      const cp = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
      return Number.isFinite(cp) ? String.fromCodePoint(cp) : m;
    }
    return ENTITIES[e.toLowerCase()] ?? m;
  });
}

/** Split off YAML frontmatter. Handles CRLF (input is normalised to \n first). */
export function splitFrontmatter(content) {
  const src = content.replace(/\r\n?/g, '\n').replace(/^/, '');
  const m = src.match(/^---[ \t]*\n([\s\S]*?)\n---[ \t]*(?:\n|$)/);
  return m ? { frontmatter: m[1], body: src.slice(m[0].length) } : { frontmatter: null, body: src };
}

/** Parse one YAML scalar (quote-aware: "a \"b\"", 'it''s', plain # comment). */
export function parseYamlScalar(raw) {
  const s = raw.trim();
  if (s.startsWith('"')) {
    const m = s.match(/^"((?:[^"\\]|\\.)*)"/);
    return m ? m[1].replace(/\\(["\\/])/g, '$1').replace(/\\n/g, '\n').replace(/\\t/g, '\t') : s.slice(1);
  }
  if (s.startsWith("'")) {
    const m = s.match(/^'((?:[^']|'')*)'/);
    return m ? m[1].replace(/''/g, "'") : s.slice(1);
  }
  return s.replace(/\s+#.*$/, '').trim();
}

/** Read a top-level frontmatter key (indented keys, e.g. openGraph.title, are ignored). Supports | and > block scalars. */
export function frontmatterValue(frontmatter, key) {
  if (!frontmatter) return null;
  const lines = frontmatter.split('\n');
  const re = new RegExp(`^${key.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}:[ \\t]*(.*)$`);
  for (let i = 0; i < lines.length; i++) {
    const m = lines[i].match(re);
    if (!m) continue;
    const rest = m[1].trim();
    if (/^[|>][-+]?$/.test(rest)) {
      const block = [];
      for (let j = i + 1; j < lines.length && (/^\s+\S/.test(lines[j]) || lines[j].trim() === ''); j++) block.push(lines[j].trim());
      return rest.startsWith('>') ? block.filter(Boolean).join(' ') : block.join('\n').trim();
    }
    if (rest === '') return null;
    return parseYamlScalar(rest);
  }
  return null;
}

// --- JS/TS scanning primitives (heuristic, not a full parser) ---

const isWordChar = (c) => c !== undefined && /[\w$]/.test(c);

/** From an opening quote at i, return the index just past the closing quote. */
export function skipString(src, i) {
  const q = src[i];
  let j = i + 1;
  while (j < src.length) {
    const c = src[j];
    if (c === '\\') { j += 2; continue; }
    if (q === '`' && c === '$' && src[j + 1] === '{') { j = matchBrace(src, j + 1) + 1; continue; }
    if (c === q) return j + 1;
    if (c === '\n' && q !== '`') return j + 1; // unterminated: stop at end of line
    j++;
  }
  return j;
}

/** From an opening bracket at i ({, ( or [), return the index of its matching closer. */
export function matchBrace(src, i) {
  const open = src[i];
  const close = { '{': '}', '(': ')', '[': ']' }[open];
  let depth = 0;
  for (let j = i; j < src.length; j++) {
    const c = src[j];
    // A quote opens a string only after a non-word character, so apostrophes in JSX text ("don't") are ignored.
    if ((c === '"' || c === "'" || c === '`') && !isWordChar(src[j - 1])) { j = skipString(src, j) - 1; continue; }
    if (c === open) depth++;
    else if (c === close) { depth--; if (depth === 0) return j; }
  }
  return src.length - 1;
}

function stripJsComments(src) {
  return src
    .replace(/\{\s*\/\*[\s\S]*?\*\/\s*\}/g, ' ') // {/* JSX comment */}
    .replace(/\/\*[\s\S]*?\*\//g, ' ')
    .replace(/^[ \t]*\/\/.*$/gm, '')
    .replace(/[ \t]+\/\/ .*$/gm, '');
}

/** Remove top-level import/export statements (multi-line aware). Used for MDX. */
export function dropModuleStatements(src) {
  let out = '';
  let i = 0;
  while (i < src.length) {
    const atLineStart = i === 0 || src[i - 1] === '\n';
    if (atLineStart && /^(import|export)\b/.test(src.slice(i, i + 7))) {
      let j = i;
      while (j < src.length && src[j] !== '\n') {
        const c = src[j];
        if (c === '"' || c === "'" || c === '`') { j = skipString(src, j); continue; }
        if (c === '{' || c === '(' || c === '[') { j = matchBrace(src, j) + 1; continue; }
        j++;
      }
      i = j;
      continue;
    }
    out += src[i++];
  }
  return out;
}

const BLOCK_TAGS = new Set(['p', 'div', 'section', 'article', 'main', 'header', 'footer', 'aside', 'nav', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'li', 'ul', 'ol', 'blockquote', 'pre', 'table', 'tr', 'td', 'th', 'thead', 'tbody', 'figure', 'figcaption', 'dl', 'dt', 'dd', 'details', 'summary', 'br', 'hr', 'form', 'label', 'button']);
const SKIP_TAGS = new Set(['script', 'style', 'noscript', 'template', 'svg', 'head', 'code', 'pre']);

function scanTag(src, i) {
  let j = i + 1;
  const closing = src[j] === '/';
  if (closing) j++;
  const nm = /^[A-Za-z][\w.:-]*/.exec(src.slice(j, j + 100));
  const name = nm ? nm[0] : '';
  j += name.length;
  while (j < src.length && src[j] !== '>') {
    const c = src[j];
    if (c === '"' || c === "'") { const k = src.indexOf(c, j + 1); j = k === -1 ? src.length : k + 1; continue; }
    if (c === '{') { j = matchBrace(src, j) + 1; continue; }
    j++;
  }
  const selfClosing = src[j - 1] === '/';
  return { end: j, name, closing, selfClosing };
}

const TAG_CONTEXT = /[([{},;:?=>&|!]/;
function isTagStart(src, i, depth) {
  const next = src[i + 1];
  if (!(next && (/[A-Za-z>]/.test(next) || next === '/'))) return false;
  if (depth > 0) return true;
  let k = i - 1;
  while (k >= 0 && /\s/.test(src[k])) k--;
  if (k < 0) return true;
  if (TAG_CONTEXT.test(src[k])) return true;
  return /\b(return|yield|default)$/.test(src.slice(Math.max(0, k - 6), k + 1));
}

/**
 * Visible text of JSX: only text inside JSX elements (depth > 0) is emitted; attributes,
 * non-literal {expressions}, generics and surrounding TS code are dropped. Block-level
 * elements become paragraph breaks.
 */
export function jsxToText(src) {
  let out = '';
  let depth = 0;
  let i = 0;
  const n = src.length;
  while (i < n) {
    const c = src[i];
    if (c === '<' && isTagStart(src, i, depth)) {
      const t = scanTag(src, i);
      const lname = t.name.toLowerCase();
      if (!t.closing && !t.selfClosing && SKIP_TAGS.has(lname)) {
        const close = src.toLowerCase().indexOf(`</${lname}`, t.end);
        const gt = close === -1 ? -1 : src.indexOf('>', close);
        i = gt === -1 ? n : gt + 1;
        out += '\n\n';
        continue;
      }
      if (t.closing) depth = Math.max(0, depth - 1);
      else if (!t.selfClosing) depth++;
      if (BLOCK_TAGS.has(lname)) out += '\n\n';
      i = t.end + 1;
      continue;
    }
    if (depth === 0) {
      if ((c === '"' || c === "'" || c === '`') && !isWordChar(src[i - 1])) { i = skipString(src, i); continue; }
      i++;
      continue;
    }
    if (c === '{') {
      const j = matchBrace(src, i);
      const inner = src.slice(i + 1, j);
      const lit = inner.match(/^\s*(["'`])([\s\S]*)\1\s*$/);
      if (lit && !(lit[1] === '`' && lit[2].includes('${'))) out += lit[2];
      else out += ` ${jsxToText(inner)} `;
      i = j + 1;
      continue;
    }
    out += /\s/.test(c) ? ' ' : c;
    i++;
  }
  return out;
}

function markdownToText(src) {
  return src
    .replace(/^(```|~~~)[\s\S]*?^\1[^\n]*$/gm, '\n')
    .replace(/<!--[\s\S]*?-->/g, ' ')
    .replace(/`[^`\n]*`/g, ' ')
    .replace(/!\[[^\]]*\]\([^)]*\)/g, ' ')
    .replace(/\[([^\]]+)\]\([^)]*\)/g, '$1')
    .replace(/\[([^\]]+)\]\[[^\]]*\]/g, '$1')
    .replace(/^[ \t]*\[[^\]]+\]:[ \t]*\S+.*$/gm, '')
    .replace(/<\/?[A-Za-z][^>]*>/g, ' ')
    .replace(/^[ \t]{0,3}#{1,6}[ \t]+/gm, '')
    .replace(/^[ \t]{0,3}>[ \t]?/gm, '')
    .replace(/^[ \t]*(?:[-*+]|\d+[.)])[ \t]+/gm, '')
    .replace(/^[ \t]*(?:[-*_][ \t]*){3,}$/gm, '')
    .replace(/(\*\*|__|~~)/g, '')
    .replace(/(^|\W)[*_]([^*_\n]+)[*_](?=\W|$)/g, '$1$2');
}

function htmlToText(src) {
  let s = src.replace(/<!--[\s\S]*?-->/g, ' ');
  const main = s.match(/<main\b[^>]*>([\s\S]*?)<\/main>/i);
  if (main) s = main[1];
  s = s.replace(/<(script|style|noscript|template|svg|head|nav|footer)\b[\s\S]*?<\/\1\s*>/gi, ' ');
  s = s.replace(new RegExp(`</?(?:${[...BLOCK_TAGS].join('|')})\\b[^>]*>`, 'gi'), '\n\n');
  s = s.replace(/<[^>]+>/g, ' ');
  return s;
}

function tidy(text) {
  return decodeEntities(text)
    .replace(/[ \t ]+/g, ' ')
    .replace(/ *\n */g, '\n')
    .replace(/\n{3,}/g, '\n\n')
    .trim();
}

/**
 * Extract the visible text of a content file. Paragraphs are separated by blank lines.
 * ext: .md .mdx .tsx .jsx .html (.htm); anything else is treated as plain text.
 */
export function extractVisibleText(content, ext) {
  const e = ext.toLowerCase();
  const src = content.replace(/\r\n?/g, '\n');
  if (e === '.tsx' || e === '.jsx' || e === '.ts' || e === '.js') return tidy(jsxToText(stripJsComments(src)));
  if (e === '.md' || e === '.mdx') {
    let body = splitFrontmatter(src).body;
    if (e === '.mdx') {
      body = dropModuleStatements(body).replace(/\{\/\*[\s\S]*?\*\/\}/g, ' ');
      let out = '';
      for (let i = 0; i < body.length; i++) {
        if (body[i] === '{') { i = matchBrace(body, i); out += ' '; continue; }
        out += body[i];
      }
      body = out;
    }
    return tidy(markdownToText(body));
  }
  if (e === '.html' || e === '.htm') return tidy(htmlToText(src));
  return tidy(src);
}

/** Count words: whitespace-separated tokens containing at least one letter or digit. */
export function countWords(text) {
  return text.split(/\s+/).filter((w) => /[\p{L}\p{N}]/u.test(w)).length;
}

/** Split extracted text into paragraphs. */
export function paragraphsOf(text) {
  return text.split(/\n\s*\n/).map((p) => p.replace(/\s+/g, ' ').trim()).filter(Boolean);
}

const DATE_KEYS = ['datePublished', 'publishedAt', 'published_at', 'published', 'pubDate', 'date', 'created', 'createdAt'];

/** Find a publish date in frontmatter, Next.js metadata / JSON-LD, or HTML meta. Returns a Date or null. */
export function findPublishDate(content) {
  const { frontmatter } = splitFrontmatter(content);
  for (const k of DATE_KEYS) {
    const v = frontmatterValue(frontmatter, k);
    if (v) { const d = new Date(v); if (!Number.isNaN(d.getTime())) return d; }
  }
  const patterns = [
    /["']?(?:datePublished|publishedTime|publishedAt|pubDate)["']?\s*:\s*["'`](\d{4}-\d{2}-\d{2}[^"'`]*)["'`]/,
    /<meta\s[^>]*property=["']article:published_time["'][^>]*content=["']([^"']+)["']/i,
    /<meta\s[^>]*content=["']([^"']+)["'][^>]*property=["']article:published_time["']/i,
    /\b(?:date|published)\s*:\s*["'`](\d{4}-\d{2}-\d{2}[^"'`]*)["'`]/,
  ];
  for (const p of patterns) {
    const m = content.match(p);
    if (m) { const d = new Date(m[1]); if (!Number.isNaN(d.getTime())) return d; }
  }
  return null;
}

/** Page age in days from a content date, falling back to file mtime. */
export function pageAge(file, content, now = new Date()) {
  const d = findPublishDate(content);
  if (d) return { ageDays: Math.floor((now - d) / 86400000), ageSource: 'content-date', date: d.toISOString().slice(0, 10) };
  const mtime = statSync(file).mtime;
  return { ageDays: Math.floor((now - mtime) / 86400000), ageSource: 'file-mtime', date: mtime.toISOString().slice(0, 10) };
}

/** Internal/external link targets written in the source: href="…", href={'…'}, Markdown [x](…) and [x]: … */
export function extractLinkTargets(content) {
  const src = content.replace(/\r\n?/g, '\n');
  const out = [];
  const patterns = [
    /\bhref\s*=\s*(?:\{\s*)?(["'`])([^"'`\n]+?)\1/g,
    /(?<!!)\[[^\]\n]*\]\(\s*<?([^)\s>]+)>?(?:\s+["'][^"']*["'])?\s*\)/g,
    /^[ \t]*\[[^\]]+\]:[ \t]*<?(\S+?)>?(?:[ \t]+["'(].*)?$/gm,
  ];
  for (const p of patterns) {
    for (const m of src.matchAll(p)) {
      const url = p === patterns[0] ? m[2] : m[1];
      if (url.includes('${')) continue;
      out.push(url);
    }
  }
  return out;
}

/**
 * Map a content file to its URL path, relative to the scanned dir.
 *   blog/foo/page.tsx → /blog/foo   (App Router; (groups) and @slots dropped)
 *   posts/foo.md      → /posts/foo  ; index.md → directory path
 * Returns null for dynamic routes ([slug]) whose URL depends on data.
 */
export function urlPathForFile(file, dir, prefix = '') {
  let rel = relative(dir, file).split(sep).join('/');
  rel = rel.replace(/^(?:src\/)?app\//, '').replace(/\.[^./]+$/, '');
  let segs = rel.split('/');
  if (['page', 'index'].includes(segs[segs.length - 1])) segs.pop();
  segs = segs.filter((s) => !/^\(.*\)$/.test(s) && !s.startsWith('@'));
  if (segs.some((s) => /^\[.*\]$/.test(s))) return null;
  return normalizePath(`${prefix.replace(/\/$/, '')}/${segs.join('/')}`);
}

export function normalizePath(p) {
  let out = p.replace(/\/{2,}/g, '/').replace(/[?#].*$/, '');
  try { out = decodeURI(out); } catch { /* keep as is */ }
  if (out.length > 1) out = out.replace(/\/$/, '');
  return out.startsWith('/') ? out : `/${out}`;
}

// ───────────────────────── thin-content analysis ─────────────────────────

/** Analyse one file. Pure except for `age`, which the caller supplies. */
export function analyzeThin({ text, content, age, minWords = 300, minAgeDays = 90 }) {
  const wordCount = countWords(text);
  const linkCount = extractLinkTargets(content).length;
  const issues = [];
  const critical = wordCount < minWords / 3;
  if (wordCount < minWords) {
    issues.push({ type: 'thin', detail: `${wordCount} visible words (minimum: ${minWords})`, severity: critical ? 'critical' : 'warning' });
  }
  if (linkCount > 10 && wordCount < Math.max(minWords, 500)) {
    issues.push({ type: 'link_heavy', detail: `${linkCount} links but only ${wordCount} words of prose (hub/list page?)`, severity: 'warning' });
  }
  let recommendation = null;
  if (issues.length) {
    const oldEnough = age && age.ageDays >= minAgeDays;
    recommendation = critical && oldEnough ? 'NOINDEX_CANDIDATE' : 'REVIEW';
    if (critical && !oldEnough) issues.push({ type: 'too_new', detail: `only ${age?.ageDays ?? '?'} days old (min-age-days: ${minAgeDays}); give it time before judging`, severity: 'info' });
  }
  return { wordCount, linkCount, issues, recommendation };
}

async function main() {
  const args = cli(import.meta.url, {
    dir: { type: 'string', required: true },
    ext: { type: 'string' },
    'min-words': { type: 'string', default: '300' },
    'min-age-days': { type: 'string', default: '90' },
    output: { type: 'string', default: 'thin-content.json' },
  });
  const minWords = Number(args['min-words']);
  const minAgeDays = Number(args['min-age-days']);
  if (!Number.isFinite(minWords) || !Number.isFinite(minAgeDays)) throw new Error('--min-words and --min-age-days must be numbers');

  const files = walkContentFiles(args.dir, parseExts(args.ext));
  console.log(`\nScanning ${files.length} files for thin content...\n`);

  const results = [];
  let skippedBinary = 0;
  let skippedDynamic = 0;
  for (const file of files) {
    if (/(^|[\\/])\[[^\]\\/]+\]([\\/]|$)/.test(file)) { skippedDynamic++; continue; }
    const content = readTextFile(file);
    if (content === null) { skippedBinary++; continue; }
    const text = extractVisibleText(content, extname(file));
    const age = pageAge(file, content);
    const a = analyzeThin({ text, content, age, minWords, minAgeDays });
    if (a.issues.length) {
      results.push({ file, wordCount: a.wordCount, linkCount: a.linkCount, ageDays: age.ageDays, ageSource: age.ageSource, issues: a.issues, recommendation: a.recommendation });
    }
  }
  results.sort((a, b) => a.wordCount - b.wordCount || a.file.localeCompare(b.file));

  const candidates = results.filter((r) => r.recommendation === 'NOINDEX_CANDIDATE');
  const review = results.filter((r) => r.recommendation === 'REVIEW');

  console.log('═'.repeat(60));
  console.log(' THIN CONTENT REPORT');
  console.log('═'.repeat(60));
  console.log(`\n  Files scanned:        ${files.length}${skippedBinary ? ` (${skippedBinary} binary skipped)` : ''}`);
  if (skippedDynamic) console.log(`  Dynamic routes:       ${skippedDynamic} skipped ([slug] pages render their text from data)`);
  console.log(`  Thin pages found:     ${results.length}`);
  console.log(`  NOINDEX_CANDIDATE:    ${candidates.length}`);
  console.log(`  REVIEW:               ${review.length}`);

  for (const tier of [
    { label: `NOINDEX_CANDIDATE: under ${Math.round(minWords / 3)} words and ${minAgeDays}+ days old. Confirm ~zero traffic in GSC first`, items: candidates },
    { label: 'REVIEW: expand, merge, or leave (short pages such as tools can rank fine)', items: review },
  ]) {
    if (tier.items.length === 0) continue;
    console.log(`\n  ${tier.label}:`);
    for (const r of tier.items.slice(0, 20)) {
      console.log(`    ${r.file} (${r.wordCount} words, ${r.ageDays}d old via ${r.ageSource})`);
      for (const issue of r.issues) console.log(`      → ${issue.detail}`);
    }
    if (tier.items.length > 20) console.log(`    ... and ${tier.items.length - 20} more`);
  }

  writeFileSync(args.output, JSON.stringify(results, null, 2) + '\n');
  console.log(`\n  Full report: ${args.output}\n`);
}

const isMain = process.argv[1] && import.meta.url === pathToFileURL(realpathSync(process.argv[1])).href;
if (isMain) main().catch(fail);

scripts/weekly-report.mjs

#!/usr/bin/env node
/**
 * Weekly SEO Performance Report
 *
 * Pulls Search Console data and writes a weekly report: week-over-week totals, top pages,
 * trending and new queries, dropping pages, CTR triage (title rewrite candidates) and
 * queries you already rank 1-3 for.
 *
 * Windows are anchored on the LAST DATE WITH FINAL DATA (not "yesterday"): GSC's final data
 * lags 2-3 days, and a week ending yesterday would be short, producing false drops.
 * "This week" = the 7 days ending on that date; "last week" = the 7 days before it.
 *
 * Usage:
 *   node scripts/weekly-report.mjs --site sc-domain:example.com [--output weekly-seo-report.json]
 *
 * Options:
 *   --site     Search Console property (required): sc-domain:example.com or https://www.example.com/
 *   --output   Output JSON path (default: weekly-seo-report.json)
 *   --help     Show this help
 *
 * Output (JSON):
 *   period             { startDate, endDate, previousStartDate, previousEndDate } for this/last week
 *   summary            thisWeek / lastWeek totals, `changes` (% change, or null when last week was 0)
 *                      and `changesFormatted` ("+9.8%" or "n/a"; use these in issue text, never "+null%")
 *   topPages           top 50 pages by clicks over 28 days (URL variants such as a trailing slash are summed)
 *   trending           queries this week vs their weekly average over the prior 3 weeks, ranked by
 *                      impressions gained. Queries with no prior impressions have `new: true` and no growthPct.
 *   dropping           pages whose clicks fell >= 30% between the two halves of the 28 days. Both halves are
 *                      fetched in full, so `absentInRecent: true` means the page really had no data.
 *   ctrTriage          ALL pages ranked by impressions with >= 1000 impressions, CTR < 2%, position 3-20
 *   topRankingQueries  queries at average position <= 3 with >= 100 impressions this week
 *
 * Exit codes: 0 report written, 1 error.
 *
 * Designed to run weekly via GitHub Actions (.github/workflows/weekly-seo-report.yml).
 */
import { writeFileSync } from 'node:fs';
import { resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { cli, fail } from '../lib/cli.mjs';
import { gscClient, queryAll } from '../lib/gsc.mjs';
import { lastDataDate, shiftDay, windowEnding, sumByPage, round, pct } from '../lib/gsc-rows.mjs';

export function pctChange(old, cur) {
  if (!old) return null;
  return round(((cur - old) / old) * 100, 1);
}
export function fmtChange(p) {
  if (p == null || !Number.isFinite(p)) return 'n/a';
  return p >= 0 ? `+${p}%` : `${p}%`;
}

/** All windows, anchored on the last final-data date. */
export function windows(end) {
  const thisWeek = windowEnding(end, 7);
  const lastWeek = windowEnding(shiftDay(end, -7), 7);
  return {
    thisWeek, lastWeek,
    last28: windowEnding(end, 28),
    prior3Weeks: { startDate: shiftDay(end, -27), endDate: shiftDay(end, -7) },
    firstHalf: { startDate: shiftDay(end, -27), endDate: shiftDay(end, -14) },
    secondHalf: windowEnding(end, 14),
  };
}

/** Totals from rows with dimensions ['date'] that fall inside a window. */
export function totals(dateRows, { startDate, endDate }) {
  let clicks = 0, impressions = 0, pw = 0;
  for (const r of dateRows) {
    const d = r.keys[0];
    if (d < startDate || d > endDate) continue;
    clicks += r.clicks; impressions += r.impressions; pw += r.position * r.impressions;
  }
  return { clicks, impressions, ctr: impressions ? clicks / impressions : 0, position: impressions ? pw / impressions : 0 };
}

export function summarize(dateRows, w) {
  const a = totals(dateRows, w.thisWeek), b = totals(dateRows, w.lastWeek);
  const view = (t) => ({ clicks: t.clicks, impressions: t.impressions, ctr: pct(t.ctr), position: round(t.position, 1) });
  const changes = {
    clicks: pctChange(b.clicks, a.clicks),
    impressions: pctChange(b.impressions, a.impressions),
    ctr: pctChange(b.ctr, a.ctr),
  };
  return {
    thisWeek: view(a), lastWeek: view(b), changes,
    changesFormatted: Object.fromEntries(Object.entries(changes).map(([k, v]) => [k, fmtChange(v)])),
  };
}

/**
 * Queries this week vs their weekly average over the prior 3 weeks. Ranked by impressions
 * gained (finite for new queries too). New queries get `new: true` and no growthPct.
 */
export function trendingQueries(thisRows, priorRows, { minImpressions = 50, minGrowthPct = 100, priorWeeks = 3, top = 20 } = {}) {
  const prior = new Map(priorRows.map((r) => [r.keys[0], r]));
  const out = [];
  for (const r of thisRows) {
    if (r.impressions < minImpressions) continue;
    const priorWeekly = (prior.get(r.keys[0])?.impressions || 0) / priorWeeks;
    const base = { query: r.keys[0], impressions: r.impressions, clicks: r.clicks, ctr: pct(r.ctr), position: round(r.position, 1), priorWeeklyImpressions: round(priorWeekly, 1), impressionDelta: round(r.impressions - priorWeekly, 1) };
    if (priorWeekly === 0) { out.push({ ...base, new: true }); continue; }
    const growthPct = round(((r.impressions - priorWeekly) / priorWeekly) * 100, 0);
    if (growthPct >= minGrowthPct) out.push({ ...base, new: false, growthPct });
  }
  return out.sort((a, b) => b.impressionDelta - a.impressionDelta || a.query.localeCompare(b.query)).slice(0, top);
}

/** firstHalf/secondHalf: Map<path, agg> from sumByPage over COMPLETE page lists. */
export function droppingPages(firstHalf, secondHalf, { minClicks = 5, minDropPct = 30, top = 15 } = {}) {
  const out = [];
  for (const [path, r] of firstHalf...