Engineer Context with Filesystem Persistence

Covers using the filesystem as a context-engineering layer so agents can store, retrieve, and update effectively unlimited context despite a fixed context window, pulling relevant context on demand (dynamic context discovery) instead of carrying everything statically. It frames four predictable context-engineering failure modes: needed context absent from the total available context, retrieved context failing to encapsulate what's needed, retrieved context far exceeding what's needed (wasting tokens, degrading performance), and agents failing to discover niche information buried across many files.

Six patterns are documented. Pattern 1, filesystem as scratch pad: write large tool outputs (for example an 8k-token web search result) to a file above a token threshold, return a file reference plus a short summary instead of the full content, and retrieve specifics later via grep or ranged reads. Pattern 2, plan persistence: write structured plans (shown as YAML with an objective, status, and per-step status) to the filesystem so the agent can re-read and re-orient at any point - "manipulating attention through recitation." Pattern 3, sub-agent communication via filesystem: sub-agents write findings directly to files (a per-agent workspace under workspace/agents/<name>/) that a coordinator reads directly, avoiding the fidelity loss of message-passing "telephone." Pattern 4, dynamic skill loading: keep only skill names and brief descriptions in static context, and load a skill's full SKILL.md file only when the task needs it. Pattern 5, terminal and log persistence: sync long-running terminal output to files and grep them for specific patterns instead of pasting full histories into context. Pattern 6, learning through self-modification: agents write learned user preferences to their own instruction files so later sessions load them automatically - flagged as an emerging pattern needing guardrails against accumulating incorrect or contradictory instructions over time.

def handle_tool_output(output: str, threshold: int = 2000) -> str:
    if len(output) < threshold:
        return output

    # Write to scratch pad
    file_path = f"scratch/{tool_name}_{timestamp}.txt"
    write_file(file_path, output)

    # Return reference instead of content
    key_summary = extract_summary(output, max_tokens=200)
    return f"[Output written to {file_path}. Summary: {key_summary}]"

Filesystem search combines ls/list_dir, glob, grep, and ranged read_file calls, which often outperforms semantic search for technical content with sparse semantic meaning but clear structural patterns; the two search modes are complementary rather than competing. A related use: when a context window limit forces summarization, writing the full conversation history to a file first lets the agent recover specific details later by searching that file instead of losing them to the summary. Stated guidelines also call for cleaning up scratch files to prevent unbounded growth and guarding self-modification patterns with validation.

Use filesystem patterns when tool outputs exceed roughly 2000 tokens, tasks span multiple turns, multiple agents need to share state, skills or instructions exceed what fits comfortably in the system prompt, or logs need selective querying.

Avoid them when tasks complete in a single turn, context already fits comfortably in the window, latency is critical since file I/O adds overhead, or the model is too simple to use filesystem tools reliably.

Recommends structuring files for discoverability (separate scratch/, memory/, skills/, and agents/ directories, consistent naming, timestamps or IDs in scratch filenames) and tracking token accounting - static-versus-dynamic context ratio, tool-output sizes before and after offloading, and how often dynamic context actually gets loaded - to optimize from measurements rather than assumptions.

Connects to context-optimization (filesystem offloading as a form of observation masking), memory-systems (filesystem-as-memory as a simple memory layer), multi-agent-patterns (file workspaces enabling agent isolation), context-compression (file references as lossless compression), and tool-design (tools should return file references for large outputs). External references include LangChain's Deep Agents work, Cursor's dynamic context discovery patterns, and Anthropic's Agent Skills specification.

Agent builders hitting context-window limits from large tool outputs, long-horizon tasks that lose track of their plan, multi-agent systems degrading through message-passing, or systems with too many skills to fit in a static system prompt.