Setup and Optimize Vector Databases

This skill covers vector database architecture, setup, and optimization for AI applications - embedding storage, similarity search, and Retrieval-Augmented Generation systems. It frames database selection around scale requirements, the embedding dimensions your model produces, latency-versus-accuracy tradeoffs, integration fit with your existing stack, and storage/compute/operational cost, and covers four index types by use case: HNSW for high-recall, moderate-scale applications; IVF for large-scale datasets with an acceptable recall tradeoff; LSH for approximate, speed-prioritized search; and flat/brute-force search for small datasets or when perfect accuracy is required.

It provides working setup configurations for three vector databases: Pinecone (index creation with a chosen embedding dimension and cosine metric, metadata indexing for filtered fields, pod-based scaling, and batched vector upserts), Weaviate (a schema-driven class definition using the text2vec-openai vectorizer and qna-openai module for built-in question answering), and Chroma (a persistent local client with a custom OpenAI embedding function and batched document adds). Performance guidance covers HNSW parameter tuning (ef and efConstruction for recall versus speed, maxConnections for recall versus memory) and query optimization strategies - pre-filtering (recommended for high-selectivity filters) versus post-filtering, and hybrid search that combines vector similarity with keyword search and re-ranks the merged results.

Production guidance covers a Docker Compose setup for local Weaviate development and a health-monitoring pattern that reports vector count, index size, p95 query latency, and memory usage. Best practices span data management (batch operations for throughput, indexing only frequently filtered metadata fields, normalizing vectors for cosine similarity, using namespaces to isolate data types), security (API key rotation, network isolation and VPCs, audit logging, rate limiting), and scalability planning (budgeting for 2-3x growth in vectors and query volume, monitoring index-build and query-latency trends, and considering multi-region deployment for global applications).

def upsert_vectors_batch(vectors_data, batch_size=100):
    for i in range(0, len(vectors_data), batch_size):
        batch = vectors_data[i:i + batch_size]
        index.upsert(vectors=batch, namespace="documents")

Use this skill when setting up or tuning a vector database for embeddings, similarity search, or RAG - choosing between Pinecone, Weaviate, and Chroma, selecting an index type for your scale and recall needs, tuning HNSW parameters, or adding hybrid search and production monitoring.

It is not a fit for choosing or training the embedding model itself - the guidance assumes you already have embeddings (or an embedding function like OpenAI's) and focuses on storing, indexing, and querying them efficiently.

Inputs are your embedding dimensions, expected vector count and query volume, and latency/accuracy requirements. Outputs are working setup code for Pinecone, Weaviate, or Chroma, tuned index configuration, pre-filtering and hybrid-search query patterns, a Docker Compose file for local development, and a health-check function reporting vector count, index size, and p95 latency.

Covers Pinecone, Weaviate, and Chroma as vector database backends, with OpenAI embedding models (text-embedding-ada-002) used as the default vectorizer in the Weaviate and Chroma setup examples.

AI engineers setting up embedding storage or RAG retrieval who need concrete, working configuration for a specific vector database rather than starting from each database's separate documentation - including index-type selection, HNSW tuning, hybrid search, and production monitoring and security practices.