Optimize Embedding Strategies for Vector Search

This skill guides selecting and optimizing embedding models for vector search. It compares six models by dimensions, max tokens, and best use: text-embedding-3-large (3072 dims, 8191 tokens, high accuracy), text-embedding-3-small (1536 dims, cost-effective), voyage-2 (1024 dims, 4000 tokens, code and legal), bge-large-en-v1.5 (1024 dims, 512 tokens, open source), all-MiniLM-L6-v2 (384 dims, 256 tokens, fast and lightweight), and multilingual-e5-large (1024 dims, 512 tokens, multi-language). It provides five template categories: OpenAI embeddings with batching and Matryoshka dimension reduction; local embeddings via sentence-transformers, including a LocalEmbedder class and an E5-style embedder that prefixes queries with query: and documents with passage:; four chunking strategies - by token count with a configurable overlap, by sentences respecting size limits, by markdown headers preserving hierarchy, and a recursive character splitter that prefers a separator inside each window; a domain-specific pipeline that preprocesses, chunks, embeds, and attaches metadata to documents for vector storage, plus a code-specific pipeline stub for tree-sitter-based function/class chunking; and a retrieval-quality evaluator computing precision@k, recall@k, MRR, and NDCG@k, plus a cosine/euclidean/dot-product similarity matrix function.

Use it when choosing embedding models for RAG, optimizing chunking strategies, fine-tuning embeddings for a domain, comparing embedding model performance, reducing embedding dimensions, or handling multilingual content. It does not apply outside embedding strategy work - a different domain or tool is out of scope.

Inputs: raw documents or queries to embed, and the target use case (RAG, code search, multilingual, low-latency). Outputs: a chosen embedding model and dimension count, chunked text via the appropriate strategy, embedding vectors (via OpenAI's API or a local sentence-transformers model), vector-storage-ready records with document ID, chunk index, text, embedding, and metadata, and retrieval-quality metrics (precision@k, recall@k, MRR, NDCG@k) plus a similarity matrix for evaluating embedding quality.

OpenAI's embeddings API (text-embedding-3-small/-large with optional dimensions reduction), sentence-transformers for local models (BGE, multilingual-E5), tiktoken for token-based chunking, nltk for sentence tokenization, tree-sitter for code parsing, and NumPy/SciPy for similarity computation. Documented external resources: the OpenAI Embeddings guide, the Sentence Transformers docs, and the MTEB benchmark leaderboard.

Document → Chunking → Preprocessing → Embedding Model → Vector
                ↓
        [Overlap, Size]  [Clean, Normalize]  [API/Local]

Engineers building RAG or vector-search systems who need to pick and optimize an embedding model, chunk documents correctly, and measure retrieval quality rather than guessing at defaults. Its best practices: match the model to the use case (code versus prose versus multilingual), chunk thoughtfully to preserve semantic boundaries, normalize embeddings for cosine similarity, batch requests instead of embedding one at a time, and cache embeddings to avoid recomputing - while avoiding ignoring token limits (truncation loses information), mixing embedding models (their vector spaces are incompatible), skipping preprocessing, or over-chunking to the point of losing context.