Implement Vector Search and Semantic Retrieval Systems

This skill covers vector database engineering across Pinecone, Weaviate, Qdrant, Milvus, and pgvector for RAG applications, recommendation systems, and similarity search. It spans vector database selection and architecture, embedding model selection and optimization, index configuration (HNSW, IVF, PQ), hybrid search combining vector similarity with keyword matching, document chunking strategies, metadata filtering (pre- and post-filtering to narrow the search space), and performance tuning/scaling to millions of vectors.

Its eight-step workflow moves from analyzing data characteristics and query patterns, through selecting an embedding model, designing a chunking/preprocessing pipeline, choosing the vector database and index type, configuring a metadata schema for filtering, implementing hybrid search where needed, optimizing the latency/recall tradeoff, and setting up monitoring and reindexing strategies. Best practices include choosing embedding dimensions appropriate to the use case (typically 384-1536), chunking with overlap rather than hard cuts, using metadata filtering to reduce search space before the vector comparison, monitoring embedding drift over time, planning for index rebuilds as data evolves, caching frequent queries, and explicitly testing recall-vs-latency tradeoffs rather than assuming a default configuration is optimal.

Use this skill proactively for vector search implementation, embedding optimization, or semantic retrieval system design - building RAG systems, implementing semantic search over documents, creating recommendation engines, building image/audio similarity search, optimizing vector search latency and recall, or scaling vector operations to millions of vectors.

It is not the right skill for tasks unrelated to vector databases and semantic search, or for projects needing a different domain or tool entirely - general database work without an embedding/similarity-search component falls outside its scope.

Input is a semantic search, RAG, or recommendation requirement - a document corpus to index, a similarity-search latency problem, or a scaling challenge as vector counts grow. Output is a vector database architecture decision, an embedding and chunking pipeline, an index configuration (HNSW/IVF/PQ) tuned for the recall/latency tradeoff, and a metadata filtering/monitoring strategy for production use.

Works across the major vector database options - Pinecone, Weaviate, Qdrant, Milvus, and pgvector (Postgres) - selecting between them based on the specific RAG/recommendation/similarity-search use case rather than defaulting to one.

Engineers building RAG systems, semantic search, recommendation engines, or similarity search over images/audio who need to choose the right vector database, embedding strategy, and index configuration rather than defaulting to naive brute-force vector comparison.