Analyze Data Distributions Accurately

This skill covers analyzing statistical distributions - identifying, fitting, testing, and visualizing probability distributions in data, spanning parametric and non-parametric methods, goodness-of-fit tests, and advanced statistical modeling techniques. Distribution-identification strategy: start with exploratory data analysis (histograms, Q-Q plots, descriptive statistics), consider the data-generating process and domain context when choosing candidate distributions, use multiple goodness-of-fit tests (Kolmogorov-Smirnov, Anderson-Darling, Shapiro-Wilk), validate with visual diagnostics and cross-validation, and account for sample-size limits on statistical power. Key distribution families: continuous (Normal, Log-normal, Exponential, Gamma, Beta, Weibull, Pareto, Student's t), discrete (Poisson, Binomial, Negative Binomial, Geometric), heavy-tailed (Cauchy, Levy, alpha-stable), and bounded (Uniform, Beta, Triangular, truncated distributions).

A comprehensive analysis workflow is demonstrated via a class handling exploratory analysis:

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import seaborn as sns
from scipy.optimize import minimize
from sklearn.preprocessing import StandardScaler

class DistributionAnalyzer:
    def __init__(self, data):
        self.data = np.array(data)
        self.results = {}
        
    def exploratory_analysis(self):
        """Comprehensive EDA for distribution analysis"""
        fig, axes = plt.subplots(2, 3, figsize=(15, 10))
        
        # Histogram with KDE
        axes[0,0].hist(self.data, bins=30, density=True, alpha=0.7)
        axes[0,0].plot(*stats.gaussian_kde(self.data).evaluate(np.linspace(self.data.min(), self.data.max(), 100)))
        axes[0,0].set_title('Distribution Shape')
        
        # Q-Q plots for normal and exponential
        stats.probplot(self.data, dist="norm", plot=axes[0,1])
        axes[0,1].set_title('Normal Q-Q Plot')
        
        stats.probplot(self.data, dist="expon", plot=axes[0,2])
        axes[0,2].set_title('Exponential Q-Q Plot')
        
        # Box plot and violin plot
        axes[1,0].boxplot(self.data)
        axes[1,0].set_title('Box Plot')
        
        axes[1,1].violinplot(self.data)
        axes[1,1].set_title('Violin Plot')
        
        # Empirical CDF
        sorted_data = np.sort(self.data)
        y_vals = np.arange(1, len(sorted_data) + 1) / len(sorted_data)
        axes[1,2].plot(sorted_data, y_vals, 'b-', linewidth=2)
        axes[1,2].set_title('Empirical CDF')
        
        plt.tight_layout()
        return self._get_descriptive_stats()
    
    def _get_descriptive_stats(self):
        return {
            'mean': np.mean(self.data),
            'std': np.std(self.data),
            'skewness': stats.skew(self.data),
            'kurtosis': stats.kurtosis(self.data),
            'cv': np.std(self.data) / np.mean(self.data) if np.mean(self.data) != 0 else np.inf
        }

and distribution fitting (fitting up to eight candidate distributions - normal, exponential, gamma, log-normal, beta, Weibull, Pareto, uniform - with bounded distributions handled specially, then computing Kolmogorov-Smirnov and Anderson-Darling statistics plus AIC/BIC, ranked by AIC).

Statistical testing and validation covers normality tests (Shapiro-Wilk, Jarque-Bera, D'Agostino K-squared), an exponentiality test (rate-parameter estimation followed by a KS test against the fitted exponential CDF), a uniformity test (KS test against uniform), and outlier detection via both the IQR method and a Z-score threshold of 3. Visualization and diagnostics cover a best-fit plotting method showing the top-N fitted distributions overlaid on the data histogram (labeled with AIC), a P-P plot for the single best fit, a residual plot (observed minus expected quantiles), and an AIC-comparison bar chart across the top distributions.

Expert recommendations match distribution families to data-generating processes: Normal for symmetric data from additive processes (central limit theorem applies), Log-normal for positive data from multiplicative processes with right skew, Exponential for time-to-event data with the memoryless property, Gamma for waiting time across multiple events on positive continuous data, Weibull for reliability analysis and hazard-function modeling, and Beta for proportions, percentages, or bounded [0,1] continuous data. Common mistakes to avoid: relying solely on visual analysis instead of quantitative tests, ignoring sample size when interpreting goodness-of-fit p-values, underestimating parameter-estimation uncertainty in small samples, failing to validate distributional assumptions on held-out data, and being careless with heavy-tailed distributions and extreme values. Advanced techniques cover mixture models for multimodal data, transformation techniques (Box-Cox, Yeo-Johnson) for better fits, truncated or censored distributions for bounded data, bootstrap methods for parameter confidence intervals, and cross-validation for model selection in predictive contexts.

Use it when identifying, fitting, or testing the probability distribution behind a dataset - choosing candidate distributions, ranking fits by AIC or BIC, running normality or goodness-of-fit tests, or diagnosing fit quality visually. It is not a general EDA or summary-statistics tool beyond distribution work - it is scoped specifically to distributional identification, fitting, and validation.

Given a numeric dataset, it produces exploratory diagnostic plots, a ranked list of fitted distributions with goodness-of-fit statistics (KS, AD, AIC, BIC), a battery of statistical tests (normality, exponentiality, uniformity, outliers), and best-fit comparison visualizations.

Code samples use numpy, pandas, scipy.stats (fitting, kstest, anderson, shapiro, jarque_bera, normaltest, zscore), matplotlib/seaborn for visualization, scipy.optimize, and scikit-learn's StandardScaler.

Data scientists and statisticians identifying, fitting, and validating the probability distribution underlying a dataset.