Implement Great Expectations Data Validation

Great Expectations Data Validation Expert helps define expectation suites, configure data contexts and checkpoints, validate batches, profile datasets, and monitor results. Examples organize expectations around completeness, uniqueness, validity, and consistency. They check row counts, ordered columns, nulls, uniqueness, regular expressions, allowed values, and statistical ranges.

The skill also shows custom expectations, checkpoint actions for storing validation results and updating Data Docs, profiling to create an initial suite, and validation in a CI/CD pipeline. It discusses organizing reusable templates, versioning suites as schemas evolve, incremental validation, caching, sampling, and choosing SQL-based execution for database sources.

Use this skill when designing data quality checks with Great Expectations and integrating validation into batch processing or a delivery workflow. It can help turn data requirements into explicit expectations and define how validation results are stored, rendered, or used to determine pipeline success.

Do not copy sample row counts, thresholds, column names, allowed statuses, regexes, Slack settings, version numbers, or performance assumptions as production rules. These need to come from the data contract and current Great Expectations APIs. It version-specific configuration patterns; verify them against the installed library version before implementation. Do not treat a profile-generated expectation as a confirmed business rule without owner review.

Inputs include the data context, datasource and data asset, batch request, expected columns and types, business rules, validation thresholds, environment-specific settings, and intended checkpoint actions. For custom expectations, define the metric dependency, inputs, success condition, and rendered explanation. For profiling, identify ignored columns and expectations that should not be inferred automatically.

The output can be an expectation suite, checkpoint configuration, validation result, profile-based starting suite, or CI/CD validation function. Results can indicate whether checks passed and include observed values. Checkpoint actions may store results, update Data Docs, or notify on failure. Separate dev, staging, and production configuration and secure credentials through appropriate secret stores.

It Great Expectations (gx), Python, Pandas execution, filesystem data connectors and stores, SQL-based execution engines, and BatchRequest. It also mentions Data Docs, Slack, email, PagerDuty, cloud secret stores, and CI/CD pipelines. Examples include batch data from CSV files and a tabular data workflow. Verify each integration and configuration field against the actual environment and library version.

This skill is for data engineers and platform teams responsible for data quality in analytics and processing pipelines. It helps make checks reusable, categorized, and observable. Data owners should review inferred expectations and thresholds before they gate production processing.

config_version: 3.0
config_variables_file_path: uncommitted/config_variables.yml

datasources:
  my_datasource:
    class_name: Datasource