Analyze Correlations and Visualize Relationships

This skill covers correlation analysis - statistical methods for measuring relationships between variables, selecting appropriate correlation measures, interpreting results, handling edge cases, and communicating findings effectively. Correlation-type selection: Pearson (linear relationships between continuous variables, assumes normality), Spearman (monotonic relationships, robust to outliers and non-normal distributions), Kendall's Tau (better for small samples, handles tied ranks well), point-biserial (continuous vs. binary variable), the phi coefficient (binary vs. binary), and Cramer's V (categorical variables with multiple levels). Data-requirements assessment: check data types and distributions before choosing a method, identify outliers that could distort Pearson correlations, assess linearity assumptions via scatter plots, and consider sample-size effects on statistical power.

Data preparation and validation is demonstrated via a function selecting numeric columns and applying appropriate checks:

import pandas as pd
import numpy as np
from scipy import stats
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.stats import pearsonr, spearmanr, kendalltau

def prepare_correlation_data(df, method='pearson'):
    """
    Prepare data for correlation analysis with appropriate checks
    """
    # Remove non-numeric columns for numeric correlations
    numeric_df = df.select_dtypes(include=[np.number])
    
    # Check for sufficient data
    if len(numeric_df) < 3:
        print("Warning: Sample size < 3, results may be unreliable")
    
    # Handle missing values
    missing_pct = numeric_df.isnull().sum() / len(numeric_df)
    high_missing = missing_pct[missing_pct > 0.1]
    if not high_missing.empty:
        print(f"Variables with >10% missing: {high_missing.index.tolist()}")
    
    # Remove outliers for Pearson (optional)
    if method == 'pearson':
        Q1 = numeric_df.quantile(0.25)
        Q3 = numeric_df.quantile(0.75)
        IQR = Q3 - Q1
        outlier_mask = ~((numeric_df < (Q1 - 1.5 * IQR)) | (numeric_df > (Q3 + 1.5 * IQR))).any(axis=1)
        clean_df = numeric_df[outlier_mask]
        print(f"Removed {len(numeric_df) - len(clean_df)} outliers")
        return clean_df
    
    return numeric_df

Comprehensive correlation analysis is shown via a function computing both Pearson and Spearman correlation matrices alongside per-pair p-values and a significance flag (p < 0.05). Advanced techniques cover partial correlation (residualizing both variables against control variables via linear regression, then correlating the residuals) and rolling correlation for time-series data over a configurable window.

Visualization best practices cover a publication-ready correlation heatmap (masking the upper triangle, annotating non-significant cells as not significant, using a diverging red-blue colormap centered on zero) and a correlation-strength distribution histogram. Interpretation guidance covers effect-size bands (|r| under 0.1 negligible, 0.1-0.3 weak, 0.3-0.5 moderate, 0.5 or above strong, 0.8 or above very strong and flagged for potential issues) and the statistical-vs-practical-significance distinction (large samples can produce statistically significant but practically meaningless correlations; always report effect sizes alongside p-values; consider confidence intervals), demonstrated via a function computing a Pearson confidence interval using Fisher's z-transformation.

Common pitfalls and fixes cover the multiple-comparisons problem (Bonferroni correction, false-discovery-rate control for exploratory analysis, hierarchical clustering to reduce dimensionality) and spurious correlations (checking for confounding variables, accounting for time-series trends, validating findings on independent datasets, using domain knowledge to assess causal plausibility). Sample-size guidance: minimum n=10 for exploratory analysis, n of 30 or more for reliable Pearson correlations, and a power-analysis approximation of n approximately 8/r-squared + 2 for 80% power to detect correlation r.

Use it when choosing a correlation method, computing correlation matrices with significance testing, controlling for confounders via partial correlation, or building publication-ready correlation visualizations. It is not a causal-inference or regression-modeling guide beyond partial correlation - correlation strength and significance are the output, not a causal model.

Given a dataset and variables of interest, it produces correlation matrices (Pearson, Spearman, or Kendall as appropriate) with p-values and significance flags, partial or rolling correlations, confidence intervals, and correlation heatmap or distribution visualizations.

Code samples use pandas, numpy, scipy.stats (pearsonr, spearmanr, kendalltau, norm.ppf), seaborn and matplotlib for visualization, and scikit-learn's LinearRegression for partial correlation.

Data scientists and analysts choosing, computing, and correctly interpreting correlation measures between variables.