Extract Text Features for ML

This is a text-feature-extraction skill for turning raw text into numerical ML features, covering four categories: statistical features (character, word, and sentence counts, average word length, type-token ratio for lexical diversity, punctuation density), linguistic features (POS-tag ratios, named-entity counts, sentiment and formality), readability scores (Flesch-Kincaid grade, Gunning Fog index), and n-gram or bag-of-words features (TF-IDF unigrams, bigram/trigram counts, character n-grams, skip-grams). It implements a TextFeatureExtractor class built on spaCy's en_core_web_sm model, with methods for basic statistics, readability metrics, and linguistic features (POS ratios plus person and organization named-entity counts), assembled into one feature matrix via a create_feature_matrix function that optionally layers in a TfidfVectorizer and an n-gram CountVectorizer. Domain-specific extractors adapt the feature set to context - for social media text:

def extract_social_features(text):
    """Features specific to social media content"""
    return {
        'hashtag_count': len(re.findall(r'#\w+', text)),
        'mention_count': len(re.findall(r'@\w+', text)),
        'url_count': len(re.findall(r'http[s]?://\S+', text)),
        'caps_ratio': sum(1 for c in text if c.isupper()) / len(text)
    }

(the source's version also tallies an emoji count via a Unicode-range regex). A parallel extractor for formal documents pulls email addresses, phone numbers, dates, and currency mentions via regex.

Use this skill when engineering text features for a downstream classification, clustering, or other ML task and you need a broader feature set than raw embeddings alone - statistical, linguistic, readability, and n-gram features combined, with domain-specific variants for social media or formal documents. It also covers feature selection (SelectKBest with an F-test, and dropping features above a 0.95 correlation threshold) and names concrete pitfalls to avoid: data leakage from features that encode future information, overfitting high-dimensional sparse features without regularization, domain shift where features don't generalize across text sources, and mishandling texts of varying length or quality. It is not an embeddings or deep-learning-representation skill - it's scoped to hand-engineered, interpretable numerical features, so it isn't the right fit for a task that specifically wants dense learned embeddings instead.

Input is raw text, a single document or a corpus; output is a feature matrix - a pandas DataFrame combining statistical, readability, and linguistic feature columns with optional TF-IDF and n-gram columns prefixed tfidf_/ngram_ per term - optionally reduced via correlation-based or statistical feature selection before being fed into a downstream model.

Built on pandas, numpy, scikit-learn (TfidfVectorizer, CountVectorizer, SelectKBest, f_classif), spacy (POS tagging and named-entity recognition), textstat (flesch_kincaid_grade, gunning_fog), and scipy.stats for correlation-based feature pruning.

Data scientists and ML engineers building text-classification, clustering, or NLP pipelines who need a broad, interpretable feature set - statistical, linguistic, readability, and n-gram - assembled and selected consistently, rather than hand-writing feature extraction per project.