Optimize Agent Memory Retrieval

This skill covers the architecture of agent memory: short-term (context window), long-term (vector stores), and the cognitive architectures that organize them, built on the CoALA framework of semantic memory (facts), episodic memory (experiences), and procedural memory (how-to knowledge). Its core principle is that memory quality is a function of retrieval quality, not storage quantity - chunking, embedding, and retrieval strategy determine whether an agent actually remembers or effectively forgets.

It compares memory frameworks (LangMem for LangGraph agents needing semantic/episodic/procedural memory types; MemGPT/Letta for OS-style hierarchical memory tiers with automatic paging; Mem0 for a user-preference/history personalization layer), vector stores (Pinecone for managed billions-scale storage at ~5ms latency and the highest cost; Qdrant, Rust-based with the best metadata filtering at ~7ms; Weaviate for hybrid search and knowledge-graph features via GraphQL at ~10ms; ChromaDB for prototyping, ~20ms p50 at 100K vectors; pgvector for teams already on Postgres, good under 1M vectors), and embedding models (OpenAI text-embedding-3-large, 3072 dimensions, $0.13/1M tokens for best quality; text-embedding-3-small, 1536 dimensions, $0.02/1M tokens; nomic-embed-text-v1.5, 768 dimensions, open-source/local; all-MiniLM-L6-v2, 384 dimensions, lowest-latency local).

It lays out patterns for chunking (fixed-size via RecursiveCharacterTextSplitter with 256-512 token general guidance; semantic chunking by similarity breakpoint; structure-aware chunking on Markdown headers; contextual chunking - prepending an LLM-generated document-context statement to each chunk before embedding, which reduces retrieval failures by 35% per Anthropic's approach; and code-specific, language-aware chunking that respects function/class boundaries), background memory formation (processing conversations asynchronously after idle rather than in real time, for higher-quality extraction without slowing interactions, plus periodic consolidation that clusters and merges similar/duplicate memories via an LLM), and memory decay (time-based archiving past a max age, or utility-based scoring combining recency, frequency, and importance, as in the MIRIX approach, to prune low-utility memories).

await memory.semantic.upsert(
    namespace="user_profile",
    key=user_id,
    content={
        "name": "Alice",
        "preferences": ["dark mode", "concise responses"],
        "expertise_level": "developer",
    }
)

Use it when designing or debugging an agent's memory system - selecting memory types, a vector store, a chunking strategy, or diagnosing why retrieval quality is poor. It catalogs seven recurring failure modes ("sharp edges") with fixes: chunking that isolates content from its context (CRITICAL - retrieved chunks don't make sense alone; fix with contextual or hierarchical chunking); chunk size mismatched to query patterns (HIGH - a default 1000-character size works for nothing specific; fix by testing sizes per content type and measuring recall); semantic search returning irrelevant results because similarity isn't relevance (HIGH - fix with metadata filtering first, hybrid semantic+keyword search, and cross-encoder reranking); old memories overriding current information since vector stores have no temporal awareness by default (HIGH - fix with time-decay scoring, updating instead of appending preferences, and explicit fact versioning); contradictory memories retrieved together (MEDIUM - fix by detecting contradictions on storage and periodically consolidating conflicting clusters); retrieved memories exceeding the context window (MEDIUM - fix with token-budgeted retrieval and dynamic top-k); and query/document embeddings from different models producing garbage similarity scores (MEDIUM - fix by tracking embedding model version in metadata and migrating old vectors on upgrade).

Hand off rather than solve within this skill when the need exceeds memory architecture itself: production vector database operations go to a data-engineer, embedding model selection or fine-tuning to an ml-engineer, knowledge graph design to a knowledge-engineer, end-to-end RAG pipeline architecture to an llm-architect, and multi-agent shared memory to multi-agent-orchestration.

Input is conversation or document content to store (routed to semantic, episodic, or procedural namespaces) or a query to retrieve against. Output is structured memory records - user profiles, timestamped episodic events, or procedural how-to steps - or, at retrieval time, a token-budgeted context assembly combining profile, recent messages, and retrieved memories. The skill also runs validation checks that flag common defects: an in-memory store in production code (ERROR, loses data on restart), a vector upsert without metadata (WARNING, blocks filtering), a query without user filtering (ERROR, risks data leakage), a hardcoded chunk size (INFO, untested), chunking without overlap (WARNING, boundary issues), pure semantic search without metadata filters (WARNING), retrieval without a result limit (WARNING, context overflow risk), embeddings stored without model-version tracking (WARNING), and document/query embeddings from different models (ERROR).

Memory frameworks LangMem, MemGPT/Letta, and Mem0; vector stores Pinecone, Qdrant, Weaviate, ChromaDB, and pgvector; embedding providers OpenAI and open-source models like nomic-embed-text-v1.5 and all-MiniLM-L6-v2. Works well alongside the autonomous-agents, multi-agent-orchestration, llm-architect, and agent-tool-builder skills.

Agent and LLM engineers designing or debugging an agent's memory system - choosing memory types, vector stores, chunking strategy, and decay policy, and avoiding the specific retrieval failure modes cataloged here.