Design and Implement Scalable Data Pipelines

This skill provides expert data pipeline architecture for scalable, reliable, cost-effective batch and streaming data processing, covering ETL/ELT, Lambda, Kappa, and Lakehouse architecture patterns, batch/streaming ingestion, workflow orchestration with Airflow/Prefect, transformation via dbt and Spark, ACID-transactional Delta Lake/Iceberg storage, data quality frameworks (Great Expectations, dbt tests), pipeline monitoring (CloudWatch/Prometheus/Grafana), and cost optimization through partitioning, lifecycle policies, and compute selection.

Its architecture design step assesses sources/volume/latency/targets and selects the right pattern (ETL: transform before load; ELT: load then transform; Lambda: batch plus speed layers; Kappa: stream-only; Lakehouse: unified), designing the flow from sources through ingestion, processing, storage, and serving with observability touchpoints throughout.

Ingestion implementation covers batch (incremental loading via watermark columns, exponential-backoff retries, schema validation with a dead letter queue for invalid records, metadata tracking like _extracted_at) and streaming (Kafka consumers with exactly-once semantics via manual offset commits within transactions, time-windowed aggregations, error handling with replay capability). Orchestration covers Airflow (task groups, XCom, SLA monitoring, incremental execution via execution_date, exponential-backoff retries) and Prefect (task caching for idempotency, parallel execution via .submit(), artifacts for visibility, configurable-delay retries).

dbt transformation separates a staging layer (incremental materialization, deduplication, late-arriving data handling) from a marts layer (dimensional models, aggregations, business logic), with tests (unique, not_null, relationships, accepted_values, custom quality tests) and source freshness checks. Its data quality framework covers Great Expectations (table/column-level checks, checkpoints, data docs, failure notifications) and dbt tests (schema tests in YAML, dbt-expectations custom tests, results tracked in metadata).

Storage strategy covers Delta Lake (ACID transactions with append/overwrite/merge, predicate-based upserts, time travel, small-file compaction with Z-order clustering, vacuum cleanup) and Apache Iceberg (partitioning/sort-order optimization, MERGE INTO upserts, snapshot isolation and time travel, binpack file compaction, snapshot expiration). Monitoring tracks records processed/failed, data size, execution time, and success rates via CloudWatch metrics and SNS alerts by severity; cost optimization covers partitioning strategy (avoiding over-partitioning below 1GB), 512MB-1GB Parquet file sizing, hot-to-cold storage lifecycle policies, and compute selection (spot for batch, on-demand for streaming, serverless for ad hoc).

Its output deliverables span architecture documentation (diagram, tech stack justification, scalability analysis, failure modes), implementation code (ingestion, dbt/Spark transformation, orchestration DAGs, storage management, data quality suites), configuration files (DAG definitions, dbt project config, Docker Compose/K8s/Terraform infrastructure, per-environment configs), monitoring/observability (metrics, alerts, dashboards, structured logging with correlation IDs), and an operations guide (deployment/rollback, troubleshooting, scaling, cost optimization, disaster recovery). Success criteria: the pipeline meets its SLA, data quality checks pass above 99%, automatic retry and alerting work, monitoring shows health/performance, documentation enables team maintenance, cost optimization cuts infrastructure costs 30-50%, schema evolves without downtime, and end-to-end data lineage is tracked.

### Batch ingestion with validation
from batch_ingestion import BatchDataIngester
from storage.delta_lake_manager import DeltaLakeManager
from data_quality.expectations_suite import DataQualityFramework

ingester = BatchDataIngester(config={})

df = ingester.extract_from_database(
    connection_string='postgresql://host:5432/db',
    query='SELECT * FROM orders',
    watermark_column='updated_at',
    last_watermark=last_run_timestamp
)

schema = {'required_fields': ['id', 'user_id'], 'dtypes': {'id': 'int64'}}
df = ingester.validate_and_clean(df, schema)

dq = DataQualityFramework()
result = dq.validate_dataframe(df, suite_name='orders_suite', data_asset_name='orders')

delta_mgr = DeltaLakeManager(storage_path='s3://lake')
delta_mgr.create_or_update_table(
    df=df,
    table_name='orders',
    partition_columns=['order_date'],
    mode='append'
)

ingester.save_dead_letter_queue('s3://lake/dlq/orders')

Use this skill when working on data pipeline architecture tasks or workflows needing guidance, best practices, or checklists - designing ETL/ELT/Lambda/Kappa/Lakehouse patterns, implementing ingestion and orchestration, building dbt/Spark transformations, or setting up data quality and monitoring.

Not for tasks unrelated to data pipeline architecture, or where a different domain or tool is needed.

Inputs: data sources, volume, latency requirements, and target storage/serving needs.

Outputs: an architecture design (pattern selection and flow diagram), implemented ingestion/orchestration/transformation/storage code, data quality suites, monitoring dashboards and alerts, and an operations guide.

Airflow, Prefect, dbt, Apache Spark, Delta Lake, Apache Iceberg, Great Expectations, Kafka, CloudWatch, Prometheus, Grafana, Terraform, Docker/Kubernetes.

Data engineers designing and implementing scalable, reliable, cost-effective batch and streaming data pipelines.