Implement Robust Data Validation Rules

This skill provides expertise in designing and implementing comprehensive data validation rules to ensure quality, integrity, and consistency across systems and processing pipelines, building frameworks that catch issues early, produce clear error messages, and maintain high data quality standards. Core validation principles cover validation layers - syntactic (format, type, structure), semantic (business rules and logic), cross-field (relationships between data elements), temporal (time-consistency checks), and external (reference-data and lookup checks) - and a fail-fast versus collect-all strategy, using fail-fast for critical structural problems while collecting all business-rule violations to give comprehensive feedback, with configurable validation modes for different use cases.

Schema-based validation covers JSON Schema validation (defining type, pattern, format, length, and range constraints per field, with required fields and additionalProperties: false to reject unexpected fields) and SQL constraint-based validation (table-level CHECK constraints for email format, future-date limits, and positive-price/valid-percentage rules, plus cross-table validation via a trigger function that raises an exception if an order's requested quantity exceeds available inventory).

The Python validation framework implements a DataValidator class supporting both fail-fast and collect-all modes, with per-field rules and cross-field rules registered separately, running all applicable rules against a data dict and aggregating errors/warnings into a ValidationResult (is_valid, errors, warnings, field_name). Reusable validation rules are implemented as composable rule factories - required() for null/blank checks, regex_pattern() for format validation with a custom error message, and numeric_range() for bounded numeric validation - each returning a standard ValidationResult so rules compose cleanly across fields.

Use this skill when designing validation rules for a data pipeline or application that needs multi-layer checks (syntactic, semantic, cross-field) with clear, actionable error reporting. It is well suited to systems ingesting structured data from multiple sources needing consistent quality enforcement. It is not meant for trivial single-field validation with no cross-field or business-rule complexity, where a simple type check suffices.

Input: the data schema/fields and the business rules that need enforcing.

Output: JSON Schema definitions, SQL constraints/triggers, and a configurable Python validation framework with reusable rules. Example reusable validation rule:

@staticmethod
def regex_pattern(pattern: str, message: str = "Invalid format") -> Callable:
    def validate(value: Any) -> ValidationResult:
        if not isinstance(value, str) or not re.match(pattern, value):
            return ValidationResult(False, [message], [])
        return ValidationResult(True, [], [])
    return validate

Works with JSON Schema (draft-07), SQL databases (CHECK constraints, PL/pgSQL triggers), and a standalone Python validation framework.

Data and backend engineers designing validation rules for data pipelines or applications, and teams that need configurable fail-fast versus collect-all validation with clear, field-attributed error messages.