Manage AI Agent Context with Session Memory

Context Engineering - Short-Term Memory Management with Sessions

AI agents often operate in long-running, multi-turn interactions, where keeping the right balance of context is critical. If too much is carried forward, the model risks distraction, inefficiency, or outright failure. If too little is preserved, the agent loses coherence.

Here, context refers to the total window of tokens (input + output) that the model can attend to at once. For GPT-5, this capacity is up to 272k input tokens and 128k output tokens but even such a large window can be overwhelmed by uncurated histories, redundant tool results, or noisy retrievals. This makes context management not just an optimization, but a necessity.

In this cookbook, we’ll explore how to manage context effectively using the Session object from the OpenAI Agents SDK, focusing on two proven context management techniques-trimming and compression-to keep agents fast, reliable, and cost-efficient.

Why Context Management Matters

• Sustained coherence across long threads - Keep the agent anchored to the latest user goal without dragging along stale details. Session-level trimming and summaries prevent “yesterday’s plan” from overriding today’s ask.
• Higher tool-call accuracy - Focused context improves function selection and argument filling, reducing retries, timeouts, and cascading failures during multi-tool runs.
• Lower latency & cost - Smaller, sharper prompts cut tokens per turn and attention load.
• Error & hallucination containment - Summaries act as “clean rooms” that correct or omit prior mistakes; trimming avoids amplifying bad facts (“context poisoning”) turn after turn.
• Easier debugging & observability - Stable summaries and bounded histories make logs comparable: you can diff summaries, attribute regressions, and reproduce failures reliably.
• Multi-issue and handoff resilience - In multi-problem chats, per-issue mini-summaries let the agent pause/resume, escalate to humans, or hand off to another agent while staying consistent.

The OpenAI Responses API includes basic memory support through built-in state and message chaining with previous_response_id.

You can continue a conversation by passing the prior response’s id as previous_response_id, or you can manage context manually by collecting outputs into a list and resubmitting them as the input for the next response.

What you don’t get is automatic memory management. That’s where the Agents SDK comes in. It provides session memory on top of Responses, so you no longer need to manually append response.output or track IDs yourself. The session becomes the memory object: you simply call session.run("...") repeatedly, and the SDK handles context length, history, and continuity-making it far easier to build coherent, multi-turn agents.

Real-World Scenario

We’ll ground the techniques in a practical example for one of the common long-running tasks, such as:

• Multi-turn Customer Service Conversations
  In extended conversations about tech products-spanning both hardware and software-customers often surface multiple issues over time. The agent must stay consistent and goal-focused while retaining only the essentials rather than hauling along every past detail.

Techniques Covered

To address these challenges, we introduce two separate concrete approaches using OpenAI Agents SDK:

• Context Trimming - dropping older turns while keeping the last N turns.

  • Pros

    • Deterministic & simple: No summarizer variability; easy to reason about state and to reproduce runs.
    • Zero added latency: No extra model calls to compress history.
    • Fidelity for recent work: Latest tool results, parameters, and edge cases stay verbatim-great for debugging.
    • Lower risk of “summary drift”: You never reinterpret or compress facts.

    Cons

    • Forgets long-range context abruptly: Important earlier constraints, IDs, or decisions can vanish once they scroll past N.
    • User experience “amnesia”: Agent can appear to “forget” promises or prior preferences midway through long sessions.
    • Wasted signal: Older turns may contain reusable knowledge (requirements, constraints) that gets dropped.
    • Token spikes still possible: If a recent turn includes huge tool payloads, your last-N can still blow up the context.

  • Best when

    • Your tasks in the conversation is indepentent from each other with non-overlapping context that does not reuqire carrying previous details further.
    • You need predictability, easy evals, and low latency (ops automations, CRM/API actions).
    • The conversation’s useful context is local (recent steps matter far more than distant history).

• Context Summarization - compressing prior messages(assistant, user, tools, etc.) into structured, shorter summaries injected into the conversation history.

  • Pros

    • Retains long-range memory compactly: Past requirements, decisions, and rationales persist beyond N.
    • Smoother UX: Agent “remembers” commitments and constraints across long sessions.
    • Cost-controlled scale: One concise summary can replace hundreds of turns.
    • Searchable anchor: A single synthetic assistant message becomes a stable “state of the world so far.”

    Cons

    • Summarization loss & bias: Details can be dropped or misweighted; subtle constraints may vanish.
    • Latency & cost spikes: Each refresh adds model work (and potentially tool-trim logic).
    • Compounding errors: If a bad fact enters the summary, it can poison future behavior (“context poisoning”).
    • Observability complexity: You must log summary prompts/outputs for auditability and evals.

  • Best when

    • You have use cases where your tasks needs context collected accross the flow such as planning/coaching, RAG-heavy analysis, policy Q&A.
    • You need continuity over long horizons and carry the important details further to solve related tasks.
    • Sessions exceed N turns but must preserve decisions, IDs, and constraints reliably.

Quick comparison

Prerequisites

Before running this cookbook, you must set up the following accounts and complete a few setup actions. These prerequisites are essential to interact with the APIs used in this project.

Step0: OpenAI Account and OPENAI_API_KEY

• Purpose:
  You need an OpenAI account to access language models and use the Agents SDK featured in this cookbook.

• Action:
  Sign up for an OpenAI account if you don’t already have one. Once you have an account, create an API key by visiting the OpenAI API Keys page.

Before running the workflow, set your environment variables:

### Your openai key
os.environ["OPENAI_API_KEY"] = "sk-proj-..."

Alternatively, you can set your OpenAI API key for use by the agents via the set_default_openai_key function by importing agents library .

from agents import set_default_openai_key
set_default_openai_key("YOUR_API_KEY")

Step1: Install the Required Libraries

Below we install the openai-agents library (OpenAI Agents SDK)

Let's test the installed libraries by defining and running an agent.

Define Agents

We can start by defining the necessary components from Agents SDK Library. Instructions added based on the use case during agent creation.

Customer Service Agent

Context Trimming

Implement Custom Session Object

We are using Session object from OpenAI Agents Python SDK. Here’s a TrimmingSession implementation that keeps only the last N turns (a “turn” = one user message and everything until the next user message-including the assistant reply and any tool calls/results). It’s in-memory and trims automatically on every write and read.

Let's define the custom session object we implemented with max_turns=3.

How to choose the right max_turns?

Determining this parameter usually requires experimentation with your conversation history. One approach is to extract the total number of turns across conversations and analyze their distribution. Another option is to use an LLM to evaluate conversations-identifying how many tasks or issues each one contains and calculating the average number of turns needed per issue.

Below, you can see how the trimming session works for max_turns=3.

What counts as a “turn”

• A turn = one user message plus everything that follows it (assistant replies, reasoning, tool calls, tool results) until the next user message.

When trimming happens

• On write: add_items(...) appends the new items, then immediately trims the stored history.
• On read: get_items(...) returns a trimmed view (so even if you bypassed a write, reads won’t leak old turns).

How it decides what to keep

1. Treat any item with role == "user" as a user message (via _is_user_msg).
2. Scan the history backwards and collect the indices of the last N user messages (max_turns).
3. Find the earliest index among those N user messages.
4. Keep everything from that index to the end; drop everything before it.

That preserves each complete turn boundary: if the earliest kept user message is at index k, you also keep all assistant/tool items that came after k.

Tiny example

History (old → new):

0: user("Hi")
1: assistant("Hello!")
2: tool_call("lookup")
3: tool_result("…")
4: user("It didn't work")
5: assistant("Try rebooting")
6: user("Rebooted, now error 42")
7: assistant("On it")

With max_turns = 2, the last two user messages are at indices 4 and 6.
Earliest of those is 4 → keep items 4..7, drop 0..3.

Why this works well

• You always keep complete turns, so the assistant retains the immediate context it needs (both the user’s last asks and the assistant/tool steps in between).
• It prevents context bloat by discarding older turns wholesale, not just messages.

Customization knobs

• Change max_turns at init.
• Adjust _is_user_msg(...) if your item schema differs.
• If you’d rather cap by message count or tokens, replace _trim_to_last_turns(...) or add a second pass that measures tokens.

Context Summarization

Once the history exceeds max_turns. It keeps the most recent N user turns intact, summarizes everything older into two synthetic messages:

• user: "Summarize the conversation we had so far."
• assistant: {generated summary}

The shadow prompt from the user to request the summarization added to keep natural flow of the conversation without confusing the chat flow between user and assistant. Final version of the generated summary injected to assistant message.

Summarization Prompt

A well-crafted summarization prompt is essential for preserving the context of a conversation, and it should always be tailored to the specific use case. Think of it like being a customer support agent handing off a case to the next agent. What concise yet critical details would they need to continue smoothly? The prompt should strike the right balance: not overloaded with unnecessary information, but not so sparse that key context is lost. Achieving this balance requires careful design and ongoing experimentation to fine-tune the level of detail.

Key Principles for Designing Memory Summarization Prompts

• Milestones: Highlight important events in the conversation-for example, when an issue is resolved, valuable information is uncovered, or all necessary details have been collected.

• Use Case Specificity: Tailor the compression prompt to the specific use case. Think about how a human would track and recall information in working memory while solving the same task.

• Contradiction Check: Ensure the summary does not conflict with itself, system instructions or tool definitions. This is especially critical for reasoning models, which are more prone to conflicts in the context.

• Timestamps & Temporal Flow: Incorporate timing of events in the summary. This helps the model reason about updates in sequence and reduces confusion when forgetting or remembering the latest memory over a timeline.

• Chunking: Organize details into categories or sections rather than long paragraphs. Structured grouping improves an LLM’s ability to understand relationships between pieces of information.

• Tool Performance Insights: Capture lessons learned from multi-turn, tool-enabled interactions-for example, noting which tools worked effectively for specific queries and why. These insights are valuable for guiding future steps.

• Guidance & Examples: Steer the summary with clear guidance. Where possible, extract concrete examples from the conversation history to make future turns more grounded and context-rich.

• Hallucination Control: Be precise in what you include. Even minor hallucinations in a summary can propagate forward, contaminating future context with inaccuracies.

• Model Choice: Select a summarizer model based on use case requirements, summary length, and tradeoffs between latency and cost. In some cases, using the same model as the AI agent itself can be advantageous.

High‑level idea

• A turn = one real user message plus everything that follows it (assistant replies, tool calls/results, etc.) until the next real user message.

• You configure two knobs:

  • context_limit: the maximum number of real user turns allowed in the raw history before we summarize.

  • keep_last_n_turns: how many of the most recent turns to keep verbatim when we do summarize.

    • Invariant: keep_last_n_turns <= context_limit.

• When the number of real user turns exceeds context_limit, the session:

  1. Summarizes everything before the earliest of the last keep_last_n_turns turn starts,

  2. Injects a synthetic user→assistant pair at the top of the kept region:

     • user: "Summarize the conversation we had so far." (shadow prompt)
     • assistant: {generated summary}

  3. Keeps the last keep_last_n_turns turns verbatim.

This guarantees the last keep_last_n_turns turns are preserved exactly as they occurred, while all earlier content is compressed into the two synthetic messages.

You can use the get_items_with_metadata method to get the full history of the session including the metadata for debugging and analysis purposes.

Notes & design choices

• Turn boundary preserved at the “fresh” side: the keep_last_n_turns user turns remain verbatim; everything older is compressed.
• Two-message summary block: easy for downstream tooling to detect or display (metadata.synthetic == True).
• Async + lock discipline: we release the lock while the (potentially slow) summarization runs; then re-check the condition before applying the summary to avoid racey merges.
• Idempotent behavior: if more messages arrive during summarization, the post-await recheck prevents stale rewrites.

Evals

Ultimately, evals is all you need for context engineering too. The key question to ask is: how do we know the model isn’t “losing context” or "confusing context"?

While a full cookbook around memory could stand on its own in the future, here are some lightweight evaluation harness ideas to start with:

• Baseline & Deltas: Continue running your core eval sets and compare before/after experiments to measure memory improvements.
• LLM-as-Judge: Use a model with a carefully designed grader prompt to evaluate summarization quality. Focus on whether it captures the most important details in the correct format.
• Transcript Replay: Re-run long conversations and measure next-turn accuracy with and without context trimming. Metrics could include exact match on entities/IDs and rubric-based scoring on reasoning quality.
• Error Regression Tracking: Watch for common failure modes-unanswered questions, dropped constraints, or unnecessary/repeated tool calls.
• Token Pressure Checks: Flag cases where token limits force dropping protected context. Log before/after token counts to detect when critical details are being pruned.