Implement LLM Operations for Production AI

LLM Operations addresses the engineering between a prototype and an operable language-model product. Its scope includes retrieval-augmented generation, embeddings, vector databases, prompt design, semantic caching, usage costs, quality evaluation, and production architecture. Fine-tuning, streaming, and agents are also named application areas.

The retrieval workflow separates document chunking, embedding generation, vector storage, query embedding, semantic search, and response generation with retrieved context. A partial Python example combines persistent Chroma storage with an Anthropic client. It validates chunk overlap, associates chunks with document metadata, and limits retrieval to available records. Real reindexing also needs a tracked inventory so stale chunks can be removed.

Vector-store selection compares Chroma for local development, pgvector for existing PostgreSQL environments, Pinecone for managed hosting, and Weaviate or Qdrant for other requirements. The SQL example illustrates an embedding table and similarity query, but schema, dimensions, indexing, and distance metric require adaptation.

Use it when an application needs specialized retrieval, embedding, evaluation, prompt, cache, or cost work. A simpler domain-specific tool is preferable when it already covers the request. The examples require compatible pinned dependencies, an available model, authorized data, and permission before context is sent to a provider. Default embedding behavior may download a model.

The example does not implement authentication or isolation between users. Personal data in vectors, scores, and retrieved context needs minimization, scope separation, and retention rules before storage or external transmission. Production DDL requires an authorized plan, and usage estimates do not authorize spending.

Inputs include documents, queries, application requirements, provider pricing, usage assumptions, and versioned reference cases. Outputs include retrieval designs, database choices, prompt structures, cache plans, cost estimates, and evaluation records. A Python setup starts with:

import chromadb
from anthropic import Anthropic

Prompt guidance separates identity, response rules, capabilities, limitations, and personalization. Results should state evidence, assumptions, sources, and limitations without requesting private internal reasoning.

Semantic caches need compatible embeddings, validated similarity, TTL, invalidation, and memory limits. Keys include tenant or user, permissions, model, configuration, and source versions. Private answers do not cross user boundaries, and side-effecting operations are not cached. False matches are measured before activation.

Cost estimates use dated verified prices and distinguish input and output usage. Embeddings, tools, retries, and storage are accounted for separately. Evaluation starts with versioned questions and observable criteria, compares reviewed references and deterministic checks, and treats model scores as probabilistic signals. Invalid JSON, refusals, and errors remain unevaluated rather than becoming passing scores.

Workflow command conventions address RAG setup, document indexing, prompt optimization, cost estimation, evaluation, caching, and model selection. These are workflow labels rather than executable tools supplied by the file.