Track Data Lineage Across Your Ecosystem

This skill covers designing and implementing systems that capture, store, and visualize data flow through complex data ecosystems - critical for compliance, debugging, impact analysis, and data governance. It works across four granularity levels: table-level lineage (relationships between source and target tables), column-level lineage (mapping transformations and dependencies of individual columns), field-level lineage (detailed tracking inside nested structures like JSON and arrays), and process-level lineage (capturing transformation logic and business rules) - collected via static analysis (parsing SQL, code, and config files), dynamic tracking (runtime instrumentation and monitoring), metadata collection (extraction from catalogs and orchestration tools), or manual annotation for complex business logic.

It provides four implementation patterns: an OpenLineage-standard client that emits run events with job, run, and dataset facets for a SQL job; a static SQL lineage parser that extracts source tables, target tables, and column references from a parsed SQL statement; an Airflow-aware operator that declares input and output tables as lineage inlets/outlets on a DAG; and a Neo4j graph-database store that models tables and jobs as nodes connected by FEEDS/PROCESSES/PRODUCES relationships, supporting upstream-table traversal to a configurable depth.

It also covers a data-quality impact analyzer that finds every downstream table affected by a quality issue in a source table and scores the severity by how many tables are affected, plus visualization and reporting guidance (visual lineage graphs, impact-analysis reports for change management, column-level lineage maps for compliance audits) and monitoring guidance (alerts for lineage breaks or anomalies, freshness and completeness metrics, automated lineage quality checks).

def _calculate_severity(self, table_count: int) -> str:
    if table_count > 20:
        return 'CRITICAL'
    elif table_count > 5:
        return 'HIGH'
    else:
        return 'MEDIUM'

Use this skill when building a lineage system that needs to survive at scale - not just emit events, but stay fast and accurate as the number of tracked tables grows: caching frequently queried lineage paths, summarizing lineage for very large datasets, and validating captured lineage against real data flows rather than trusting the collector blindly.

It is not a fit for general data-catalog or metadata-management design beyond lineage itself - the guidance is scoped to capturing and querying data flow relationships, not to the broader catalog features around them.

Beyond the four implementation patterns above, this skill's best practices cover two output types not yet mentioned: business-facing data-journey documentation (a plain-language narrative of how data moves, distinct from the column-level compliance-audit maps) and comprehensive logging and error handling wrapped around the lineage-tracking system itself, so failures in the tracker are visible rather than silently producing incomplete lineage.

OpenLineage supplies the event schema that both the Airflow operator and any custom job emit into; Neo4j is the storage layer those events (or statically-parsed SQL lineage) ultimately land in, queried via upstream-traversal Cypher rather than relational joins; and existing data catalogs and database audit trails are named as additional metadata sources to hook into for automatic, rather than manually-annotated, lineage capture.

Data engineers and platform teams building data lineage or governance tooling who care about lineage staying trustworthy over time, not just about wiring up the initial capture - validating accuracy through sampling and cross-verification against real data flows, and configuring lineage-quality checks so gaps get caught rather than silently accumulating.