Encode Categorical Data for Machine Learning

Categorical Variable Encoding Expert guides selecting and implementing categorical-encoding strategies for machine learning: one-hot and dummy encoding for low-cardinality columns, target, frequency, and binary encoding for medium cardinality, hash encoding and embeddings for high cardinality, and ordinal encoding for ordered categories - along with data-leakage prevention and production handling of unseen categories.

Use it when preparing categorical features for a model - choosing an encoding based on column cardinality and model type, implementing cross-validated target encoding without leaking the target, reducing dimensionality with binary or embedding-based encoding for a high-cardinality column, or building a production-safe encoder that handles categories unseen at training time. It is not a general feature-engineering or model-selection agent - it's scoped specifically to converting categorical columns into numeric features, and its own guidance defers other choices, like which model to use, to model-specific recommendations rather than making them.

Input is a categorical column or columns, its cardinality, and whether a target variable is available for supervised encoding. Output is Python code, mostly scikit-learn and pandas: one-hot encoding via pd.get_dummies or OneHotEncoder with unknown-category handling; a target_encode_cv function that computes Bayesian-smoothed, K-fold cross-validated target means to avoid overfitting; category_encoders.BinaryEncoder for high-cardinality columns, reducing 100 categories to 7 binary features; frequency- and count-encoding functions that map unseen test categories to zero; a TruncatedSVD-based embedding function built on top of one-hot encoding; a multi_encode_categorical function that generates frequency, count, target, and ordinal encodings for the same column at once; a RobustCategoricalEncoder class with fit/transform that falls back to a stored mean for target encoding of unseen categories; and a validate_encoding diagnostic function reporting dimensionality, memory usage, introduced nulls, and the feature-expansion ratio.

Built on scikit-learn (OneHotEncoder, LabelEncoder, TruncatedSVD, KFold), pandas, and the category_encoders library for binary encoding:

def frequency_encode(train_series, test_series=None):
    freq_map = train_series.value_counts(normalize=True).to_dict()
    train_encoded = train_series.map(freq_map)
    
    if test_series is not None:
        test_encoded = test_series.map(freq_map).fillna(0)  # Unknown categories get 0
        return train_encoded, test_encoded
    return train_encoded

Data scientists and ML engineers preparing categorical features for training who need the encoding matched to both cardinality and model family - one-hot or avoiding high-cardinality ordinal for linear models, ordinal, target, or frequency encoding for tree models, and embedding layers for neural networks on high-cardinality columns - and who need production-safe handling of categories the encoder never saw during training.