Build Advanced Ensemble Machine Learning Models

This skill covers ML ensemble methods - combining multiple models for superior predictive performance - spanning voting classifiers, bagging, boosting, stacking, and advanced ensemble architectures, grounded in bias-variance tradeoffs and model-diversity principles. Diversity requirements covered: algorithm diversity (combining fundamentally different algorithms - tree-based, linear, neural), data diversity (different subsets/features/representations of training data), hyperparameter diversity (varying model configs to capture different patterns), and training diversity (different random seeds, CV folds, or bootstrap samples). Bias-variance guidance: pair high-bias models (linear) with low-bias models (trees) for a balanced ensemble; bagging reduces variance, boosting reduces bias; stacking learns optimal combination weights.

Voting ensembles are demonstrated with scikit-learn's VotingClassifier/VotingRegressor, combining diverse base models (logistic regression, random forest, SVM with probability=True) into a soft-voting ensemble with per-model weights, evaluated against individual models via cross_val_score. Advanced stacking is covered through an AdvancedStacker class that generates out-of-fold meta-features via k-fold cross-validation (cloning and refitting each base model per fold to avoid leakage), then trains a meta-model on those meta-features:

from sklearn.linear_model import Ridge
from xgboost import XGBClassifier

base_models = [
    ('rf', RandomForestClassifier(n_estimators=100, max_depth=5)),
    ('xgb', XGBClassifier(n_estimators=100, max_depth=3)),
    ('lr', LogisticRegression(C=0.1))
]

Dynamic ensemble selection is covered via a DynamicEnsemble class that uses k-nearest-neighbors to estimate each base model's local competence (accuracy in the neighborhood of a query point) and weights predictions accordingly at inference time.

Ensemble-weight optimization is shown via optimize_ensemble_weights, which uses scipy.optimize.minimize to find weights that minimize validation log-loss (or maximize accuracy), constrained to sum to 1 and stay non-negative. Diagnostics are covered via ensemble_diagnostics, which computes AUC/log-loss/accuracy per base model plus a pairwise prediction-correlation matrix - lower average correlation signals more diversity.

Best practices: start with voting ensembles before advanced stacking; validate diversity by keeping base-model correlation under 0.7; always use proper cross-validation for stacking to prevent overfitting; engineer different feature sets per base model; and balance model complexity against training-time budget. Pitfalls to avoid: data leakage (never use test data when building the ensemble), overfitting (too many base models or an overly complex meta-learner), redundant models (many similar models erode the diversity benefit), and unbalanced weights (some models dominating the ensemble). Production considerations: version base models and ensemble weights, consider parallel inference for independent base models, manage memory carefully for large ensembles, and A/B test the ensemble against the best individual model.

Use it when combining multiple trained models into a voting, stacking, or dynamically-selected ensemble to improve predictive performance beyond any single model. It is not a guide to training the individual base models themselves (feature engineering, hyperparameter tuning per algorithm) - it assumes reasonable base models exist and focuses on how to combine them.

Given a set of trained base models and validation data, it returns ensemble code (voting, stacking, or dynamic selection), optimized combination weights, and diagnostics - per-model AUC/log-loss/accuracy plus a pairwise correlation matrix to check diversity.

Code samples run on scikit-learn (VotingClassifier, RandomForestClassifier, LogisticRegression, SVC, KFold, NearestNeighbors), xgboost (XGBClassifier), scipy.optimize for weight optimization, and pandas/numpy for correlation diagnostics.

ML engineers and data scientists building ensemble models who need to combine several trained models for better predictive performance than any one model alone.