Optimize Embedding Strategies for Vector Search

Embedding Strategies guides selecting and optimizing embedding models for vector search applications. It opens with a comparison table of six embedding models by dimensions, max tokens, and best use case - text-embedding-3-large (3072-dim, high accuracy), text-embedding-3-small (1536-dim, cost-effective), voyage-2 (1024-dim, code/legal), bge-large-en-v1.5 (1024-dim, open source), all-MiniLM-L6-v2 (384-dim, fast/lightweight), and multilingual-e5-large (1024-dim, multi-language) - and a document-to-vector pipeline diagram covering chunking, preprocessing, embedding model, and vector output. Five code templates cover the full workflow: OpenAI embeddings with batched requests and Matryoshka dimension reduction:

### Dimension reduction with OpenAI
def get_reduced_embedding(text: str, dimensions: int = 512) -> List[float]:
    """Get embedding with reduced dimensions (Matryoshka)."""
    return get_embedding(
        text,
        model="text-embedding-3-small",
        dimensions=dimensions
    )

local embeddings via Sentence Transformers (a LocalEmbedder class with BGE query-prefix handling and a separate E5Embedder using "query:"/"passage:" instruction prefixes); four chunking strategies (by token count via tiktoken, by sentence via nltk respecting size limits, by semantic markdown-header sections, and a LangChain-style recursive character splitter with overlap); a DomainEmbeddingPipeline that preprocesses, chunks, embeds, and assembles vector-store records with metadata, plus a specialized CodeEmbeddingPipeline that embeds code chunks with surrounding context; and embedding quality evaluation functions computing precision@k, recall@k, MRR, and NDCG@k for retrieval, plus cosine, euclidean, and dot-product similarity matrices.

Use this skill when choosing embedding models for RAG, optimizing chunking strategies, fine-tuning embeddings for a domain, comparing embedding model performance, reducing embedding dimensions, or handling multilingual content. It is not for tasks unrelated to embedding strategies or for work in a different domain or tool outside this scope.

Given a vector-search or RAG task, the skill outputs runnable embedding and chunking code matched to the use case, plus a retrieval-quality evaluation harness (precision@k, recall@k, MRR, NDCG@k) for comparing model or chunking choices empirically rather than by guesswork.

Named models and libraries include OpenAI's text-embedding-3-large/small, Voyage's voyage-2 and voyage-code-2, BAAI's bge-large-en-v1.5, all-MiniLM-L6-v2, and intfloat's multilingual-e5-large, implemented via the openai SDK, sentence-transformers, tiktoken, nltk, and tree-sitter for code parsing. Points to the OpenAI Embeddings docs, the Sentence Transformers site, and the MTEB benchmark leaderboard for further model comparison. Five do's - match model to use case, chunk thoughtfully to preserve semantic boundaries, normalize embeddings for cosine similarity, batch requests, and cache embeddings - and four don'ts - don't ignore token limits, don't mix embedding models between incompatible vector spaces, don't skip preprocessing, and don't over-chunk and lose context - are stated explicitly.

Engineers building or tuning RAG and vector-search systems who need to choose the right embedding model, chunking strategy, and dimension trade-off for their domain, and to measure retrieval quality objectively, rather than picking a model by default and hoping it works.