Start a coding task with an AI agent. Let it inspect the repository, read logs, discuss three approaches, change direction twice, run the tests, and debug the failure.
By turn 40, the transcript contains everything — except a clear picture of what matters now.
Conversation compaction fixes that. It replaces old, noisy history with a smaller representation that preserves the state required to continue the task: goals, decisions, constraints, facts, artifacts, and open work.
This guide explains the main compaction techniques, where each one fails, and a practical architecture you can use for long-running agents.
Compaction Is Not Just Summarization
A conversation is an append-only event log. The model receives some portion of that log on every turn. As it grows, three things happen:
- Cost rises because more input tokens are processed.
- Latency rises because the model has more context to ingest.
- Signal density falls because resolved discussions compete with current instructions.
Remember! 💡 Compaction is a state-management operation.
💡 Pro tip: Optimize for recoverability, not compression ratio. If the agent can explain the current goal, the decisions already made, and the next step after compaction, the result is useful.
Five Techniques You Can Combine
| Technique | What it does | Best for | Main risk |
|---|---|---|---|
| Sliding window | Keeps only the newest messages | Stateless chat, simple assistants | Silently loses old requirements |
| Rolling summary | Replaces the whole conversation with a summary (It can be a good idea to keep latest turns too) | General conversations | Summary drift |
| Structured state | Extracts facts into a fixed schema | Agents and workflows | Schema misses unexpected details |
| Retrieval-backed memory | Stores history externally and recalls relevant parts | Large histories and research | Retrieval misses or returns noise |
| Prompt compression | Removes low-information text or tokens | Large documents and tool output | Damaged code, JSON, or meaning |
These are layers, not competitors. A robust system usually combines at least three.
1. Sliding window
Keep the system instructions and the most recent N messages. Drop everything in the middle.
This is cheap and deterministic. It is also brutally lossy. LangChain’s own trimming example shows the failure clearly: once the message containing a user’s name is removed, the agent can no longer answer a later question about it.
Use a sliding window for freshness, not memory.
2. Rolling summary
When the conversation crosses a token threshold, summarize the oldest section and keep the latest turns verbatim (Optional):
[system instructions]
[summary of turns 1–36]
[turns 37–44 verbatim]
[new user message]
This preserves narrative continuity at a fraction of the size. The weakness is recursive loss: if you repeatedly summarize a summary, small errors become durable facts.
3. Structured state
For task-oriented agents, prose is not enough. Extract the state into fields with explicit semantics:
{
"goal": "Add conversation compaction to the support agent",
"constraints": ["Keep the last 12 messages verbatim"],
"decisions": [
{
"decision": "Use a structured summary plus retrieval",
"reason": "We need both continuity and source recovery"
}
],
"artifacts": ["src/agent/memory.ts"],
"openQuestions": ["Which embedding model should index old turns?"],
"nextActions": ["Add compaction evaluation cases"]
}
Now you can validate the output, merge fields deliberately, and render only the sections needed by the next call.
⚠️ Warning: Never let the summary overwrite higher-priority instructions. System rules, security boundaries, and user constraints should remain authoritative inputs, not mutable memories.
4. Retrieval-backed memory
Do not force every tool result, document, and decision into the active prompt. Store the original content outside the conversation, then retrieve relevant fragments when needed.
A useful split is:
- Working memory: recent turns and current task state.
- Episodic memory: previous actions and their outcomes.
- Semantic memory: stable facts, preferences, and domain knowledge.
- Artifact store: files, logs, diffs, URLs, and full tool outputs.
The compacted state should point to evidence rather than paraphrase all of it. Tests failed in run test-184 is cheaper and safer than embedding 600 lines of output, as long as the agent can retrieve test-184 later.
5. Prompt compression
Prompt compression removes redundant sentences or low-information tokens before the context reaches the main model. Microsoft’s LLMLingua research reported up to 20× compression with little performance loss in its evaluated tasks. LongLLMLingua made compression question-aware and reported 1.4×–2.6× end-to-end acceleration for roughly 10K-token prompts compressed by 2×–6×.
Those results are promising, but prompt compression solves a different problem from conversation state. It can reduce bulky natural language. It should not be trusted to rewrite source code, signed reasoning blocks, exact quotations, tool-call arguments, or machine-readable data.
A Practical Compaction Architecture
The diagram below shows how each technique transforms growing conversation history into a compact working set. Trigger compaction before the context is full — a threshold around 60–75% of the usable input budget is a sensible starting point.
flowchart LR
T(["Token\nthreshold\nreached"])
T --> SW
T --> RS
T --> SS
T --> RM
T --> PC
subgraph SW ["① Sliding Window"]
SW1["Full history"] -->|"Drop old"| SW2["Latest N\nmessages"]
end
subgraph RS ["② Rolling Summary"]
RS1["Old turns"] -->|"Summarize"| RS2["Prose summary\n+ recent tail"]
end
subgraph SS ["③ Structured State"]
SS1["Conversation"] -->|"Extract"| SS2["{goal · constraints\ndecisions · openWork}"]
end
subgraph RM ["④ Retrieval Memory"]
RM1["History +\ntool output"] -->|"Archive + embed"| RM2["Working memory\n+ evidence pointers"]
end
subgraph PC ["⑤ Prompt Compression"]
PC1["Verbose text"] -->|"Compress"| PC2["Dense text\n~20× smaller"]
end
If you use OpenAI’s Responses API, the platform now supports server-side and standalone compaction. Server-side compaction can run automatically after a configured token threshold and returns an encrypted compaction item that can be passed into later requests. That removes some orchestration work, but you should still evaluate whether your application-specific constraints survive the process.