Implement Great Expectations Data Validation

This skill builds data validation systems with Great Expectations (GX), the Python data quality library. It covers Data Context configuration (great_expectations.yml with datasources, execution engines like PandasExecutionEngine, inferred and configured data connectors, and expectations/validations stores backed by filesystem stores), and comprehensive expectation-suite design grouping checks by data-quality dimension: table-level (ExpectTableRowCountToBeBetween, ExpectTableColumnsToMatchOrderedList), completeness (ExpectColumnValuesToNotBeNull), uniqueness (ExpectColumnValuesToBeUnique), validity (ExpectColumnValuesToMatchRegex for email format, ExpectColumnValuesToBeInSet for enum-like status columns), and consistency (ExpectColumnMeanToBeBetween).

suite.add_expectation(
    gx.expectations.ExpectColumnValuesToMatchRegex(
        column="email",
        regex=r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$',
        meta={"dimension": "Validity"}
    )
)

Advanced patterns cover writing custom Expectation subclasses (declaring metric_dependencies, implementing _validate() against a compliance threshold, and a prescriptive renderer for human-readable output) and production checkpoint configuration with an action_list chaining StoreValidationResultAction, UpdateDataDocsAction, and a SlackNotificationAction that fires on failure via a webhook. Automated data profiling uses UserConfigurableProfiler against a BatchRequest-scoped validator to auto-generate an initial suite, with excluded expectation types and ignored columns configurable.

CI/CD integration wraps a checkpoint run in a validate_data_pipeline() function that raises on validation failure for pipeline gating. Best practices cover organizing expectations by quality dimension with meta tags, sampling large datasets via BatchRequest row limits, incremental validation for time-series data, and SQL-based execution engines for database sources. Monitoring guidance covers multi-channel notifications (Slack, email, PagerDuty), validation-result trending, and multi-environment setups with environment-specific thresholds and secure credential management.

Use this skill when building or extending automated data quality validation with Great Expectations - defining expectation suites for completeness/uniqueness/validity/consistency checks, writing custom business-rule expectations, configuring checkpoints with failure notifications, auto-profiling new data assets, or gating a CI/CD pipeline on data validation results.

It is not the right tool for one-off manual data spot-checks (a quick pandas .isnull().sum() or SQL query is faster), or for validation needs outside GX's scope like schema migration testing or application-level input validation, which call for different tooling.

Input: the datasource(s) to validate (filesystem, database, or runtime batches) and the data quality rules needed (completeness, uniqueness, format validity, statistical bounds, or custom business rules). Output: a Data Context configuration, one or more expectation suites (built manually or via automated profiling), optional custom Expectation classes, a checkpoint configuration with a notification action list, and a CI/CD-callable validation function that fails the pipeline on unmet expectations.

Built on the Great Expectations Python library, with execution engines for pandas and SQL-based sources, filesystem-backed expectations/validations stores, and checkpoint actions integrating with Slack (via SlackNotificationAction/webhook), email, and PagerDuty for alerting. Data Docs sites render validation results for local or hosted viewing.

Data engineers and analytics engineers building automated data quality gates - particularly teams that need expectation suites organized by quality dimension, custom business-rule validation, and CI/CD-integrated checkpoints with failure alerting across multiple environments.