Build Deequ Data Quality Pipelines

This skill provides expertise in Amazon's Deequ data quality framework, specializing in implementing robust data validation pipelines, defining constraints, and detecting anomalies using Apache Spark, with deep knowledge of Deequ's analyzers, checks, verification suites, and profiling capabilities. Core Deequ principles cover constraint-based validation (declarative constraints data must satisfy), incremental computation (Spark's distributed computing for scalable quality checks), metric computation (analyzers computing statistics and quality metrics), repository-based anomaly detection (for time-series data quality), and profiling (automatic comprehensive data profiling reports).

Basic verification suite setup uses VerificationSuite().onData(df).addCheck(...) with checks like hasSize, isComplete, isUnique, isContainedIn, satisfies for SQL-expression rules, and hasPattern for regex validation (e.g. email format). Advanced constraint patterns cover statistical constraints (hasMin, hasMax, hasMean, hasStandardDeviation, hasApproxCountDistinct) and conditional business-rule constraints via satisfies with CASE WHEN expressions (e.g. shipped orders must have tracking numbers, discount cannot exceed 50% of total).

Repository-based anomaly detection uses an InMemoryMetricsRepository to save AnalyzerContext results from analyzers like Size, Completeness, Uniqueness, Mean, and StandardDeviation, then runs VerificationSuite with addAnomalyCheck using an AbsoluteChangeStrategy (e.g. flagging a change outside a plus-or-minus 10% threshold) against the stored metric history. Data profiling and constraint suggestions use ColumnProfilerRunner to generate per-column profiles (completeness, distinct count, mean, standard deviation for numeric columns) and ConstraintSuggestionRunner with Rules.DEFAULT to automatically suggest constraints with confidence scores.

Error handling and reporting processes VerificationResult, checking CheckStatus.Success and iterating checkResults/constraintResults to report failed constraints and extract metrics from successful ones for monitoring. Best practices cover using useRepository() for incremental validation over time, choosing Error level for critical business rules versus Warning for monitoring, composing multiple simple constraints rather than complex SQL expressions, caching DataFrames before multiple verification runs, implementing custom analyzers for domain-specific checks, and integrating Deequ checks into Spark applications, Airflow DAGs, or Glue jobs. Configuration management demonstrates building checks dynamically from a QualityConfig/CheckConfig structure, mapping constraint type strings to Deequ constraint methods.

Use this skill when building data quality validation pipelines on Apache Spark - defining constraints, detecting anomalies against historical metrics, or generating data profiles and constraint suggestions with Deequ. It is well suited to Spark-based data pipelines processing large-scale tabular data needing declarative, scalable quality checks. It is not meant for non-Spark data validation needs, or for datasets small enough that simpler validation libraries would suffice without Spark's distributed computation.

Input: a Spark DataFrame and the data quality constraints, anomaly detection thresholds, or profiling requirements to implement.

Output: Deequ verification suites, constraint definitions, anomaly detection configuration, and profiling/constraint-suggestion code. Example verification suite:

VerificationSuite()
  .onData(df)
  .addCheck(
    Check(CheckLevel.Error, "Data Integrity Checks")
      .hasSize(_ >= 1000)
      .isComplete("customer_id")
      .isUnique("customer_id")
      .isContainedIn("status", Array("active", "inactive", "pending"))
  )
  .run()

Builds on Amazon Deequ and Apache Spark; integrates quality checks into Spark applications, Airflow DAGs, or AWS Glue jobs, and can persist metrics via an in-memory or external metrics repository.

Data engineers building data quality pipelines on Apache Spark, and teams that need declarative constraint validation, historical anomaly detection, and automatic profiling rather than ad hoc data checks.