Setup and Optimize Vector Databases

Vector Database Setup Expert guides storage and retrieval decisions for embeddings, similarity search, and retrieval-augmented generation. Selection begins with expected vector volume, query throughput, embedding dimensions, latency and recall requirements, existing infrastructure, and storage, compute, and operational costs.

It compares HNSW, IVF, LSH, and flat search as different index choices. HNSW emphasizes high recall with memory and construction tradeoffs; IVF trades recall for larger-scale partitioned search; LSH prioritizes approximate retrieval speed. Flat or brute-force search fits smaller datasets or exact comparison needs. These are planning directions rather than performance measurements for a particular workload.

Use the skill when choosing a vector store, configuring an index or collection, designing metadata filters, or planning retrieval operations and monitoring. It gives concrete setup sketches for Pinecone, Weaviate, and Chroma.

Check the installed SDK, server, model, and deployment versions before using their APIs. The configurations contain illustrative keys, model names, and version-specific settings rather than a ready deployment. The Weaviate Compose example exposes HTTP with anonymous access enabled, so it does not establish a protected production service.

The hybrid-search sketch makes separate embedding and query-text calls, then invokes an undefined combine_results helper. It needs an implemented combination strategy and verified retrieval semantics; naming a text query does not itself demonstrate a keyword-ranking implementation.

Inputs include model dimensions, vector identifiers, documents, metadata, filtering requirements, deployment constraints, and expected growth. Outputs can include an index or collection configuration, batched ingestion functions, filtering choices, tuning settings, and an operational plan.

Pinecone setup illustrates dimension and metric selection, indexed metadata, a documents namespace, and batched upserts. Weaviate defines a Document class with content, title, category, and model-backed modules. Chroma combines persistence settings, an embedding function, a cosine-distance collection, and batched document additions. Its collection setup includes:

    metadata={"hnsw:space": "cosine"}

Batch sizing needs to fit the selected service rather than become a universal constant. Index only frequently filtered metadata, maintain vector normalization appropriate to the chosen metric, and use namespaces to separate data types.

HNSW tuning covers query exploration, construction effort, connections, vector caching, and cleanup intervals. Increasing these controls can trade query speed, index build time, memory, and recall. Selective metadata filtering and top-k limits shape the retrieval workload.

Docker Compose illustrates local Weaviate development with persistent storage and external model credentials. Production planning adds network isolation or VPCs, API-key rotation, audit logging, rate limiting, and horizontal or multi-region scaling where needed.

The skill serves developers planning embedding stores and retrieval systems. Monitoring sketches track vector count, index size, memory, latency, and unhealthy connection failures. They contain client-specific and placeholder helpers, so actual probes must match the chosen database. Growth planning also considers index build times and query-volume trends rather than relying only on initial capacity.