Monitor Database Migrations in Real-Time

This skill builds observability infrastructure for database migrations: real-time Change Data Capture (CDC) pipelines, Prometheus metrics collection, anomaly detection, and multi-channel alerting. For MongoDB, an ObservableAtlasMigration class wraps each migration in a session transaction while a Winston logger and Prometheus Histogram/Counter metrics (migration duration, documents processed, migration errors, each labeled by version and status) track progress. For CDC, a CDCObservabilityManager consumes a database.changes Kafka topic, tracks events-processed, consumer-lag, and replication-lag metrics, and configures a Debezium PostgresConnector (via connector.class: io.debezium.connector.postgresql.PostgresConnector, plugin.name: pgoutput, and a heartbeat interval) against the Kafka Connect REST API. Each parsed CDC event carries a source database, table, operation, and timestamp, and is both counted in the events-processed metric and applied to the target as it arrives.

Use it when a database migration - MongoDB, SQL via CDC, or NoSQL - needs real-time visibility into progress, lag, and errors rather than a fire-and-forget run, especially for zero-downtime migrations where proactive alerting on anomalies matters more than a post-hoc log review. It composes with three related plugins rather than replacing them: sql-migrations, nosql-migrations, and migration-integration for cross-workflow coordination.

An EnterpriseMigrationMonitor polls progress every 30 seconds and flags anomalies against fixed thresholds:

if stats.rows_per_second < stats.expected_rows_per_second * 0.5:
    anomalies.append({
        'type': 'low_throughput',
        'severity': 'warning',
        'message': f'Throughput below expected'
    })

if stats.error_rate > 0.01:
    anomalies.append({
        'type': 'high_error_rate',
        'severity': 'critical',
        'message': f'Error rate exceeds threshold'
    })

Detected anomalies route through an AlertingSystem to Slack (colored by severity: danger/warning/good) and email. A Grafana dashboard is provisioned programmatically with panels for migration progress (rate(migration_rows_total[5m])), data lag (with green/yellow/red thresholds at 0/60/300 seconds), and error rate. A GitHub Actions workflow starts monitoring before the migration runs and checks migration health afterward with a --max-lag 300 threshold, tying the CI job's pass/fail to whether replication lag stayed within that bound rather than just to the migration script's own exit code.

Builds on Prometheus (metrics), Kafka and Debezium (CDC), Grafana (dashboards via its HTTP API), Slack webhooks and email (alerting), and GitHub Actions (CI/CD); designed to plug into the sql-migrations and nosql-migrations skills for the actual migration logic while this skill supplies the observability layer.

Teams running production database migrations who need real-time lag, throughput, and error visibility with automated anomaly detection and alerting - rather than discovering a stalled or failing migration only after the fact.