Encode Categorical Data for Machine Learning

This skill is an expert in encoding categorical variables for machine learning and data analysis, covering the mathematical foundations, implementation details, and appropriate use cases for each technique. Selection is driven by cardinality: low cardinality (under 10 categories) suits one-hot or dummy encoding; medium cardinality (10-50) suits target encoding, frequency encoding, or binary encoding; high cardinality (over 50) suits hash encoding, embedding layers, or dimensionality reduction; and ordinal relationships suit ordinal encoding or a custom mapping. It is equally focused on preventing data leakage: fit encoders only on training data, apply transformations to validation/test sets separately, use cross-validation for target-based encodings, and explicitly handle unseen categories in production.

Use it when categorical features need encoding for a model and the right technique depends on cardinality, whether there's an ordinal relationship, and leakage risk - not a one-size-fits-all pd.get_dummies() call.

from sklearn.preprocessing import OneHotEncoder
import pandas as pd

### For sklearn with proper handling
encoder = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
X_train_encoded = encoder.fit_transform(X_train[['category_col']])
X_test_encoded = encoder.transform(X_test[['category_col']])
feature_names = encoder.get_feature_names_out(['category_col'])

Beyond one-hot, it implements target encoding with k-fold cross-validation and Bayesian smoothing to prevent overfitting, binary encoding via category_encoders.BinaryEncoder for high-cardinality columns (100 categories become 7 binary features instead of 100 one-hot columns), frequency and count encoding as simple value-count mappings with unseen categories filled to zero, embedding-based encoding (one-hot followed by TruncatedSVD dimensionality reduction), and a multi-encoding helper that stacks frequency, count, target, and ordinal encodings for the same column at once. For production, it provides a RobustCategoricalEncoder class that fits per-column encoders and falls back to a stored mean/mode value for categories unseen at inference. Model-specific guidance: tree-based models work well with ordinal, target, or frequency encoding; linear models want one-hot and should avoid high-cardinality ordinal encoding; neural networks favor embedding layers for high cardinality; distance-based models need the encoded features standardized afterward. A validation helper reports dimensionality before/after, memory usage, null values introduced, and the feature-expansion ratio.

Data scientists and ML engineers choosing and implementing a categorical-encoding strategy who need cardinality-appropriate technique selection, leakage-safe implementation (especially for target encoding), and a production-ready fallback for categories not seen during training. For memory-constrained or large-scale work it also covers sparse one-hot output, batch processing for large datasets, hash encoding as a memory-bounded alternative, and using pandas' native category dtype instead of plain object columns.