Generate Comprehensive Data Profiling Reports

This skill provides expertise in creating comprehensive data profiling reports that give deep insight into data quality, structure, and characteristics, combining statistical analysis, anomaly detection, and translation of technical findings into practical business recommendations. Core profiling dimensions cover completeness analysis (missing-value percentages and patterns, systematic gaps in data collection, temporal completeness trends, and cross-column dependency impact), uniqueness assessment (duplicate and near-duplicate record detection, primary key violation analysis, cardinality ratio calculation, and fuzzy-matching opportunities), and validity/consistency (business-rule and constraint validation, format consistency checks for dates/emails/phone numbers, statistical outlier detection, and cross-referencing lookup values).

The statistical profiling framework implements a DataProfiler class generating a per-column profile - dtype, null count/percentage, unique count/ratio for every column, then branching into numeric-specific statistics (mean, median, std, min/max, quartiles, skewness, kurtosis, and IQR-based outlier detection with bounds and outlier percentage) or string-specific statistics (average/min/max length, common patterns, and top value frequencies). SQL-based profiling covers a comprehensive table profiling template combining table-level stats (row count, unique keys, date range), column completeness (non-null count and percentage, uniqueness percentage), and data quality flags (regex-based invalid email/phone detection, future-dated records).

Report structure and templates define an executive summary format - dataset size, an overall quality score based on completeness/validity/consistency, critical high-priority issues, and top-3 recommended actions - followed by key findings sections on data completeness (percentage complete, critically missing columns, identified temporal gaps) and prioritized data quality issues (high/medium/low priority, each with the percentage of records affected).

Use this skill when producing a data profiling report that needs statistical rigor (outlier detection, distribution stats) combined with a business-readable executive summary and prioritized issue list. It is well suited to new dataset onboarding, data quality audits, or pre-migration assessments. It is not meant for a dataset already well understood with no profiling question to answer, or for a trivial dataset too small for meaningful statistical profiling.

Input: the dataset (table or DataFrame) to profile.

Output: a per-column statistical profile, SQL-based table-level quality flags, and an executive summary report with prioritized findings. Example numeric column profile fields:

profile.update({
    'mean': col_data.mean(),
    'std': col_data.std(),
    'skewness': stats.skew(col_data.dropna()),
    'outliers_iqr': self._detect_outliers_iqr(col_data)
})

Works with Python's pandas, numpy, and scipy.stats for statistical profiling, and SQL for table-level quality flag queries.

Data engineers and analysts producing data profiling reports for new dataset onboarding or quality audits, and teams that need a business-readable executive summary alongside the statistical detail.