Build Scalable Data Pipelines and Architectures

Data Engineer guides the design and implementation of scalable data pipelines and analytics infrastructure. It starts with sources, service-level requirements, data contracts, scale, latency, and consistency. Architecture selection then connects storage, processing, orchestration, ingestion, transformation, and validation. Monitoring, cost control, error handling, lineage, and operational runbooks are part of the implementation workflow.

Architecture coverage includes lakehouses with Delta Lake, Apache Iceberg, and Apache Hudi; cloud warehouses; organized data lakes; domain-owned data mesh; real-time analytics; and OLAP engines. Batch work includes Apache Spark 4.0, its Catalyst engine and columnar processing, dbt transformations with version control and testing, Airflow dependencies, and Python or Scala processing. Streaming work addresses change data capture, windowing, aggregations, joins, schema compatibility, event-driven architecture, and real-time feature engineering.

Data modeling includes star and snowflake schemas, data vaults, One Big Table approaches, slowly changing dimensions, incremental loading, archiving, and retention. Performance work examines partitioning, clustering, indexing, materialized views, caching, compression, distributed parallelism, bottlenecks, resource allocation, autoscaling, and spot instances. These decisions are evaluated alongside cloud workload cost and operational simplicity.

Quality and governance encompass validators, catalogs, metadata management, lineage, impact analysis, masking, anonymization, access control, row-level security, schema evolution, and backward compatibility. Security topics include encryption in transit and at rest, IAM, network and VPC configuration, audit logging, sensitivity labeling, differential privacy, k-anonymity, secure sharing, and policy enforcement.

Use this skill to design batch or streaming pipelines, build warehouses or lakehouses, or implement data quality, lineage, and governance. Example tasks include synchronizing databases through CDC, creating a multi-tenant platform with isolation, implementing domain-specific data products, and handling late-arriving or out-of-order events.

It is outside scope for exploratory analysis alone or ML model development without pipelines. Access to sources and storage systems is required. Protect personally identifiable information, apply least-privilege access, and validate data before writing to production sinks. Clarify missing inputs, permissions, boundaries, or success criteria, and retain environment-specific testing and expert review.

Inputs are data requirements, source systems, storage access, SLAs, contracts, and constraints on scale, latency, consistency, cost, and security. Outputs include architecture decisions, implemented ingestion and transformation pipelines, validation checks, monitoring and alerting, documented schemas and business logic, data-flow documentation, and maintenance runbooks. Failure recovery, backup, disaster recovery, and development, staging, and production environment management belong to the operating design.

Modern-stack options include Fivetran or Airbyte with dbt, Snowflake or BigQuery, and BI tools. Processing and discovery tools include Databricks, pandas, Polars, Ray, Great Expectations, Apache Atlas, DataHub, and Amundsen. Streaming technologies include Kafka, Confluent, Pulsar, Flink, Kafka Streams, Kinesis, Event Hubs, and Pub/Sub. Orchestration includes Prefect, Dagster, Data Factory, Step Functions, Kubernetes CronJobs, and Argo Workflows.

AWS coverage includes S3 lifecycle policies, Glue schema discovery, Redshift and Spectrum, EMR, Lake Formation, Athena, and DataBrew. Azure coverage includes Data Lake Storage Gen2, Synapse, Databricks, Stream Analytics, Purview, SQL Database, Cosmos DB, and Power BI. GCP coverage includes Cloud Storage, BigQuery, Dataflow, Composer, Data Fusion, Dataproc, and Looker. Real-time analytics options include Pinot, ClickHouse, and Druid; querying includes Presto/Trino and Spark SQL.

Database integration spans PostgreSQL, MySQL, SQL Server, MongoDB, Cassandra, DynamoDB, InfluxDB, TimescaleDB, Neo4j, Neptune, Elasticsearch, OpenSearch, Pinecone, and Qdrant. Infrastructure options include Terraform, CloudFormation, Bicep, Docker, Kubernetes, Prometheus, Grafana, and ELK. Access patterns cover REST, GraphQL, WebSockets, Server-Sent Events, gateways, rate limiting, queues, SaaS sources, synchronization, and conflict resolution.

The skill serves data engineers building maintainable data platforms and analytics infrastructure. Its approach emphasizes reliability, reproducible infrastructure, clear lineage, governance from design onward, and operationally sustainable performance.