Implement Robust Data Validation Rules

Data Validation Rules Expert helps design validation layers for data formats, business logic, relationships between fields, time consistency, and external reference values. It explains fail-fast handling for critical structural errors and collect-all behavior for business rule violations. Validation modes can be configurable for different uses.

The source demonstrates JSON Schema constraints, SQL checks and triggers, and a Python validation framework with field-level and cross-field rules. Examples include required values, regular expressions, numeric and date ranges, consistency between order totals and quantities, delivery date ordering, inventory warnings, and completeness, uniqueness, and range checks on Spark DataFrames. A YAML configuration example describes table fields and business rules.

Use this skill when defining repeatable data quality checks and deciding where they belong: schema boundary, field, cross-field business rule, database, or pipeline. It helps make validation behavior and errors consistent across processing stages. Choose fail-fast for structural problems that prevent further interpretation, and collect-all when users benefit from seeing multiple business rule violations together.

Do not adopt sample limits, regular expressions, SQL rules, thresholds, business policies, or date windows without confirming them with the data owner. Examples such as the age range, email format, discount rule, completeness threshold, and inventory policy illustrate techniques only. A warning is distinct from an invalid result; define severity and handling explicitly.

Inputs include data schemas, expected types and formats, required fields, business constraints, reference data, temporal rules, pipeline volumes, and the desired error strategy. Define expected versus actual values and stable error codes where downstream handling requires them. Consider edge cases, valid and invalid samples, boundary values, and cross-system behavior.

The output can be a schema, database constraint, reusable validation rule set, pipeline checks, or configuration file. Results may contain validity, field names, errors, and warnings. The Python example represents results with is_valid, errors, warnings, and an optional field name. Monitoring can count validations and failures by rule and derive a failure rate. Use batching operations, indexing reference lookups, and caching compiled patterns and rules when performance matters.

The material names JSON Schema, SQL, PostgreSQL-style PL/pgSQL triggers, Python, Apache Spark DataFrames, and YAML configuration. It uses Python typing, dataclasses, datetime, and regular expressions, and shows Spark functions for null and range checks. It also recommends version control, automated testing, and deployment pipelines for validation rules. These are examples;

This skill is for data engineers, application developers, and data governance teams responsible for data quality and integrity. It helps teams communicate what is checked, where failures are reported, and whether violations stop processing or produce warnings. Keep rules aligned with confirmed business policy and test boundaries before deployment.

@dataclass
class ValidationResult:
    is_valid: bool
    errors: List[str]
    warnings: List[str]
    field_name: Optional[str] = None