Generate and Manage Text Embeddings

This skill is expert in creating, managing, and optimizing text embeddings for machine learning and semantic search applications, with deep knowledge of embedding models, vector databases, similarity metrics, and best practices for high-quality vector representations. Its model selection principles call for task-specific models (search, clustering, classification each favor different tradeoffs), balancing embedding quality against computational efficiency via dimensionality, confirming language/domain support, and matching context window to typical text length. Its quality optimization covers text preprocessing and normalization for consistent embeddings, chunking strategies that preserve semantic meaning across long documents, batch processing to reduce API calls and latency, and L2 normalization for cosine similarity calculations. It covers implementation patterns including basic single and batch embedding generation via the OpenAI client, intelligent paragraph-first text chunking with token estimation and sentence-level fallback for oversized paragraphs, ChromaDB vector store integration with cosine-space collections, custom similarity functions using scipy/sklearn cosine similarity with ranked results, K-means semantic clustering, disk-based caching keyed on an MD5 hash of the input text, and input validation that rejects non-string input, strips excess whitespace, and truncates oversized text.

Use this skill when building semantic search, clustering, or classification systems that need production-grade embedding generation - not a one-off call to an embeddings API. It provides model configuration guidance by use case: text-embedding-3-large at 3072 dimensions for high-quality semantic search and cross-lingual understanding, and text-embedding-3-small at 1536 dimensions for fast clustering and classification. It bakes in production concerns from the start - caching to avoid regenerating embeddings for identical text, validation to catch NaN values and dimension mismatches, and metadata enrichment (timestamp, source, processing version) stored alongside embeddings. It is not a guide for choosing between vector database products in depth - it demonstrates ChromaDB specifically as the persistence layer, with the underlying embedding-generation and chunking logic being the reusable part.

import openai
import numpy as np
from typing import List, Dict, Any

class EmbeddingGenerator:
    def __init__(self, model_name: str = "text-embedding-3-small"):
        self.model_name = model_name
        self.client = openai.OpenAI()
    
    def generate_embedding(self, text: str) -> List[float]:
        """Generate embedding for a single text."""
        response = self.client.embeddings.create(
            input=text.strip(),
            model=self.model_name
        )
        return response.data[0].embedding
    
    def generate_batch_embeddings(self, texts: List[str]) -> List[List[float]]:
        """Generate embeddings for multiple texts efficiently."""
        cleaned_texts = [text.strip() for text in texts if text.strip()]
        
        response = self.client.embeddings.create(
            input=cleaned_texts,
            model=self.model_name
        )
        
        return [data.embedding for data in response.data]

Given raw text, the skill produces an EmbeddingGenerator class like the one above for single and batch embedding calls, an intelligent chunking function that splits by paragraph with token-aware overlap and sentence-level fallback, a ChromaDB VectorStore wrapper for adding documents and running similarity search, ranked cosine-similarity results and K-means cluster assignments, a caching decorator keyed on text hash, and input-validation/preprocessing functions with truncation for oversized text.

ML and search engineers building semantic search, RAG, or clustering pipelines who need production-ready embedding generation - chunking, caching, validation, and vector-store integration - rather than a bare API call. It suits teams that want use-case-matched model selection (search versus clustering versus multilingual) and quality-assurance discipline: consistent preprocessing, NaN/dimension validation, meaningful similarity thresholds, regular evaluation against known similar/dissimilar text pairs, and version tracking so embeddings get regenerated when the underlying model changes.