Build Robust Scikit-learn ML Pipelines

This skill builds robust, production-ready scikit-learn ML pipelines using Pipeline and ColumnTransformer. It follows data-leakage-safe design (transformers fit only on training data), reproducibility via random_state, and modular composition. It implements column transformers handling numeric, categorical, and ordinal features differently, custom BaseEstimator/TransformerMixin feature engineers, an advanced feature pipeline combining polynomial features, SelectKBest feature selection, and PCA, a full pipeline with GridSearchCV hyperparameter tuning, pipeline persistence with versioned metadata via joblib, a custom ValidationTransformer that raises on insufficient samples or excessive missing data, memory-cached pipelines for expensive operations, and debugging utilities to inspect feature names and shapes at each pipeline step.

Use this skill when building an sklearn ML pipeline - handling mixed numeric/categorical/ordinal columns with a ColumnTransformer, writing a custom transformer for domain-specific feature engineering, combining preprocessing with feature selection and PCA, tuning hyperparameters across the whole pipeline with GridSearchCV, persisting a trained pipeline with versioned metadata, adding a validation step that fails fast on bad input data, or debugging a pipeline's intermediate shapes and feature names.

It does not cover deep learning pipelines (that's a separate PyTorch/TensorFlow concern) or distributed training - it is focused on scikit-learn's Pipeline/ColumnTransformer composition model for classical ML.

Inputs are typically a mixed-type tabular dataset and a target model. Outputs include a column transformer, for example:

numeric_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler())
])

categorical_transformer = Pipeline([
    ('imputer', SimpleImputer(strategy='constant', fill_value='missing')),
    ('onehot', OneHotEncoder(drop='first', sparse=False))
])

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numeric_transformer, numeric_features),
        ('cat', categorical_transformer, categorical_features)
    ],
    remainder='drop'  # or 'passthrough' to keep other columns
)

Other outputs include a custom feature-engineering transformer, a full feature pipeline with PCA and feature selection, a GridSearchCV-tuned end-to-end pipeline, a save_pipeline/load_pipeline pair with JSON metadata, a ValidationTransformer raising on data quality issues, a memory-cached pipeline using tempfile.mkdtemp, and pipeline inspection utilities printing shape/columns at each step.

Best practices summarized at the end: always use Pipeline even for simple preprocessing to prevent data leakage, version-control pipelines with metadata, validate inputs to catch data quality issues early, use ColumnTransformer for mixed data types, cache expensive operations via the memory parameter, unit test custom transformers separately, monitor feature drift in production, and handle categorical features with appropriate encoders that account for unseen categories.

ML engineers and data scientists building production sklearn pipelines who need leakage-safe, versioned, and debuggable workflows rather than ad-hoc preprocessing scripts glued together with manual train/test handling.