Abstract. When a long-running agent session outgrows its context window, the harness summarises it and continues. This is now standard practice across the major agent tools, and it is usually described as routine housekeeping. We measured what it actually does to the knowledge inside a session. On a real working session, a single harness compaction preserved 66% of the load-bearing items we had catalogued in advance, but the loss was strongly uneven by provenance: knowledge the operator typed survived at 71%, while knowledge the agent had acquired through tool use, the largest single share of session content, survived at 44%. The sharper result is what happens on the second compaction. Because the summariser now summarises its own previous summary, standing instructions collapsed from 13/13 preserved to 4/13, and to 2/13 by the sixth compaction. Listing the user's messages, the mechanism that protects instructions on the first pass, works exactly once.
We report the measurement chain in full, including the metric's limitations, the fixture confounds we registered before reading per-item data, and our own extraction system's first score on real prose (17.6%, after 96–100% on synthetic benches), which we publish as a measured instance of fixture overfit rather than conceal. A sealed two-arm comparator has been pre-registered to replace the proxy figures reported here.
01The setting: why compaction exists, and what it is trusted with
An agent session accumulates. Every file read, every command output, every reply joins the conversation, and the whole prefix is re-processed on every subsequent call. Two pressures follow. The first is economic: input spend grows with the square of session length, a property we quantify separately[13]. The second is hard: the window is finite. When a session approaches its limit, the harness compacts: it asks a model to summarise the conversation, replaces the history with the summary, and continues.
Compaction is universal and largely undocumented in its effects. Of the eight major agent tools whose context handling we surveyed, five perform automatic summarise-and-replace compaction (Claude Code, Cline, GitHub Copilot CLI, Codex CLI, Aider); none of the eight reports to the user what a given compaction preserved or dropped[8,9,10]. Anthropic's own guidance is candid about the mechanism's limits: material that must survive reliably should be written to disk, because the summary is lossy by design[10]. The question this notebook asks is narrow: how lossy, of what, and does the loss compound?
The distinction that matters throughout is provenance. A session's knowledge arrives three ways: the operator types it, the agent reasons it into its own narrative, or it enters through tool output (file contents, command results, API responses). In our estate these areas are far from equal in size. Tool output is the largest at 46% of transcript characters; operator-typed content, including injected notifications, and agent narrative make up the rest. Whatever compaction does, it does most of it to material the operator never wrote.
02Method
The substrate is real. We drew on 2,538 working sessions from our own estate (each ≥20 turns), within which 120 sessions compact at least once and 61 compact repeatedly. That marathon tail is where the question has teeth. For depth we selected one heavily-compacted session (six harness summaries over ~850k tokens of transcript) and curated a ground-truth manifest against a fixed window of it: 102 load-bearing items across five types (standing directives, facts, learnings, procedural instructions, and time-anchored state), each item verified to be a literal substring of the window, and each attributed to its source area by locating it in the originating message class. Directives are verbatim operator turns and carry high confidence; the remaining classes are pattern-selected prose and carry the confidence of curation, not construction.
Survival is scored two ways. Verbatim: the item's text appears in the summary. Gist: at least 60% of the item's content words appear, a deliberately generous lexical proxy for semantic survival. The proxy is the method's main weakness and we treat it as such: it can credit a summary for scattered vocabulary, and it cannot credit a faithful paraphrase that changes wording. Both failure directions are possible; the second is why our sealed comparator (§11) replaces this proxy with a calibrated semantic matcher before any number here is used in a product claim.
Curation happened before any summary was read, and the diagnosis hypotheses for our own extractor's failure (§07) were registered before per-item results were opened. We have tried, wherever the order of operations could flatter us, to fix the order first.
03One compaction, measured
The first harness summary in the deep-dive session runs to roughly 5,900 tokens, the same budget class as a purpose-built salient frame. Of the 79 manifest items present in the span it summarised, 52 survived at gist level: 66%.
| by type | survived | rate | by source area | survived | rate |
|---|---|---|---|---|---|
| directive | 13/13 | 100% (verbatim) | operator-typed | 34/48 | 71% |
| instruction | 13/22 | 59% | agent narrative | 14/22 | 64% |
| fact | 10/15 | 67% | tool output | 4/9 | 44% |
| learning | 9/15 | 60% | |||
| clock / state | 7/14 | 50% |
Two readings coexist here and both are correct. The first: vanilla compaction is genuinely good at the thing its template protects. A user's instructions, on the first pass, survive verbatim; any critique of harness compaction that claims otherwise is wrong on the evidence, and we retired our own such claim when this measurement landed. The second: the protection is a template artefact, not a property of the mechanism. Nothing enumerates what the agent learned. The 34% that dies in one pass is drawn disproportionately from the areas no template lists.
04The compounding result
What does the second compaction summarise? Not the session, which is gone. It summarises the first summary, plus whatever accumulated since. The mechanism that preserved the operator's instructions on the first pass has nothing to enumerate on the second: the messages it would list are no longer messages, they are lines inside a summary. We tracked the same 79-item cohort through all six of the session's summaries.
| compaction | #1 | #2 | #3 | #4 | #5 | #6 |
|---|---|---|---|---|---|---|
| all items (of 79) | 52 | 23 | 27 | 18 | 21 | 18 |
| directives (of 13) | 13 | 4 | 5 | 4 | 2 | 2 |
An honest confound belongs here rather than in a footnote. Some late-summary absence is legitimate: as work moves on, old items lose relevance, and a summariser that drops them is doing its job. This is why the cohort's standing directives are the clean signal: a standing order ("never do X in this repository") is precisely the class that must not decay with topic drift, and it is the class that collapses at hop two. Relevance decay explains the tail of the curve; it does not explain the cliff at its start.
Independent internal studies corroborate both the cliff and its behavioural consequence. In a thirty-session controlled study with objective grading, rule-violation rates moved from 2.2% before a forced compaction boundary to 82.8% after it, with per-rule survival into the summary mediating the effect (Fisher p<0.00001) and a caution attached: survival is not restoration, since a rule present in the summary is not thereby acted on. A separate bench found adherence to a realistic instruction file perfect for roughly thirty turns and 39% in the first post-compaction bin. The attention companion paper[14] carries these results and the measured remedies.
05Where knowledge lives
The provenance finding deserves its own statement because it inverts a common intuition. Sessions are usually discussed as if their content were the conversation: what the user asked, what the model said. By volume, ours are not. Across the estate, tool output is 46% of transcript characters; in the deep-dive session it is the plurality of the manifest's factual content. The session's largest knowledge store is the one the summary template never enumerates and preserves worst, at 44%.
The practical consequence is quietly corrosive. What dies at 44% is not what the operator said. It is what the agent found out: the actual contents of the config file, the real error text, the measured result of the experiment just run. After a compaction the agent still knows it was told to fix the bug; it has forgotten what it learned about the bug. The observable behaviour (re-reading files it already read, re-running commands it already ran) is familiar to anyone who has driven a long session, and our reference-distance data suggest most re-reads reach back fewer than fifty turns, exactly the horizon a compaction erases.
06What the field does instead
Three coping strategies are observable at scale, none of which addresses the mechanism.
Restart and re-teach. In public conversation corpora, heavy users run many short sessions rather than one long one; the median heavy user in a WildChat-derived study starts 8–14 separate sessions[4]. Restarting sidesteps compaction by paying its cost up front: every session begins ignorant.
Externalise what must survive. The vendor-endorsed pattern (rules files, memory files, disk-backed notes) works, and is candidly framed by Anthropic as the only reliable path: what must survive compaction should not live in the conversation[10]. It is also a tax on the operator, who must predict in advance which knowledge will matter, and it does nothing for the 46% that arrives through tools mid-session.
Bolt on a memory database. The dedicated-memory products fare worst in controlled evaluation: on MemoryArena's interdependent multi-session tasks, the field scores under 25% and the memory products score below a plain long-context baseline[7]. A retrieval store attached to a leaking context does not stop the leak; it adds a second place for the answer to be missed. MemGPT stands apart architecturally, treating context as a managed memory hierarchy with the agent participating in its own eviction decisions[5], and is the nearest neighbour to the approach we pursue; forgetting-curve designs such as MemoryBank[6] model decay elegantly, but our field data suggest the dominant losses are not decay-shaped. They are cliff-shaped, and they happen at compaction.
07Our own attempt, reported honestly
The obvious response to §03–05 is typed extraction: parse the window into explicit units (directives, facts, learnings, state), carry directives verbatim through every rebuild, and report per-type survival on every compaction. We built this. On synthetic benches it performs to specification: 96.4% survival on a 250-item seeded population, 100% across all three source areas on a purpose-built bench, directives 50/50 verbatim at population scale.
On its first run against real prose, the same real window used throughout this notebook, it survived 18 of 102 items. 17.6%.
We publish this number for two reasons. The first is the obvious one: it was measured, and the alternative to publishing measured results is publishing selected ones. The second is that the gap itself is the finding. A system can score 96–100% on every synthetic bench its builders devise and still fail three-quarters of a real window; the benches measured our fixtures, not the problem. Real-substrate evaluation was a ruling imposed on this programme from outside the engineering loop, over the objection of convenience, and it is the only reason this failure was caught before a customer, rather than by one. On the same real cohort, the vanilla harness summary's 66% currently exceeds our extractor's 17.6% — different metrics, but the direction is not in our favour, and we state it. The claim that survives tonight is not that our extraction is better; it is that ours is measured, typed, and diagnosable, and that the multi-hop collapse of §04 is a property of summarise-your-own-summary that typed carry is structurally positioned to avoid, as the demo-scale result (a seeded directive surviving 3/3 successive rebuilds verbatim) already shows. Whether it does so at population scale on real text is exactly what the fix, and the sealed re-run, must demonstrate.
08Related work
[1] Zhao et al., WildChat: 1M ChatGPT Interaction Logs in the Wild, 2024 — conversation-length distributions for consumer chat.
[2] Zheng et al., LMSYS-Chat-1M, 2024 — large-scale real conversation corpus; mean 2.0 turns.
[3] Early ChatGPT User Portrait through the Lens of Data, 2023 — power-user tail: mean 16.5 turns, max 998.
[4] Priming, Path-dependence and Plasticity, 2026 — session-frequency behaviour; the restart pattern.
[5] Packer et al., MemGPT: Towards LLMs as Operating Systems, 2023 — context as managed memory hierarchy; nearest architectural neighbour.
[6] Zhong et al., MemoryBank, 2023 — Ebbinghaus-style forgetting for agent memory.
[7] MemoryArena benchmark — interdependent multi-session tasks; field <25%, memory products below plain context.
[8] GitHub Copilot CLI context management — documented auto-compaction at ~80% capacity.
[9] Cline Auto Compact — always-on summarise-and-replace.
[10] Anthropic memory-tool guidance — "the only things that survive compaction reliably are things that live on disk."
[11] AgentFold, 2025 and Context as a Tool, 2025 — harness-side folding for long-horizon agents.
[12] Jeffrey, Graph-Guided Context Reduction for AI Coding Agents, Cogmeta Research 2026 — the input-side counterpart: scored context injection; its Fig. 2a records a compaction event (166K→30K) as an unexamined step.
[14] Koherium, Present but Unfired, 2026 — the attention companion: the cliff, the levers, and the honest hook comparison.
[13] Jeffrey, Bug-Class Forcing Functions, Cogmeta Research 2026 — commit-time contracts versus discovery-time advice; independently converges on the structural-enforcement principle this programme measured as 28–0.
[15] Laban et al., LLMs Get Lost in Multi-Turn Conversation, 2025 — an average 39% multi-turn performance drop across six tasks, with models failing to recover after a wrong turn; behavioural corroboration of the boundary effects measured here.
[16] Jiang et al., LLMLingua, 2023 and Ge et al., In-Context Autoencoder, 2023 — the compression literature: both treat information loss as the explicit price of ratio, where harness compaction treats it as free.
[17] Hsieh et al., RULER, 2024 — effective context is far shorter than claimed context even before any compaction; loss begins while the text is still present.
[18] Zhang et al., H2O, Xiao et al., StreamingLLM, Li et al., SnapKV — KV-cache eviction: the model-layer analogue of the application-layer eviction problem studied here, with the same core question of what may be dropped.
09Limitations
The metric is a proxy. Gist survival is lexical containment, not semantic equivalence. It can flatter a summary and it can miss a paraphrase. The sealed comparator replaces it; until then every survival figure here is a proxy figure and is labelled so where it appears.
Depth over breadth. The multi-hop result is one session followed carefully, not a population statistic. The estate census establishes that multi-compaction sessions are common; it does not yet establish the decay curve's distribution across them. Extending the six-hop tracking to the full 120-session compacting cohort is mechanical and queued.
Curation confounds, registered. The manifest's non-directive items are pattern-selected prose fragments; their shape may disadvantage any extractor, including ours. This is registered as a named hypothesis in the §07 diagnosis, not discovered after it. The window itself was pre-scrubbed for secrets before fixture use, which production windows are not; interactions between scrub artefacts and extraction are likewise registered.
Single harness, single model family. All summaries are Claude Code's compaction on Sonnet-class models. The template-artefact mechanism (§04) predicts similar behaviour wherever compaction summarises its own prior summary, but we have not measured other harnesses.
What we cannot see. Session records tell us what survived in text. They do not tell us what the model could still recover by re-derivation, nor what the user silently re-taught. Both would soften the practical impact of loss; neither restores a standing order that no longer exists anywhere in context.
10Claims and non-claims
- A single harness compaction, measured on a real session, lost a third of pre-catalogued load-bearing knowledge — unevenly, with tool-derived knowledge worst at 44% survival.
- Instruction preservation through compaction is a template artefact that operates exactly once; by the second compaction 69% of standing directives were gone, 85% by the sixth.
- Tool output is the largest knowledge area in real agent sessions (46% of content) and the worst preserved.
- Synthetic extraction benches can report 96–100% while the same system scores 17.6% on real prose; real-substrate evaluation is not optional.
- That vanilla compaction fails at single-hop instruction preservation — it succeeds, verbatim, and we retired our contrary claim when the measurement said so.
- That our typed-extraction alternative currently outperforms vanilla on real text — today it does not, and the number is above.
- That the proxy metric's percentages are exact — they are bounds pending the sealed semantic comparator.
- That the multi-hop collapse generalises beyond this harness — predicted by mechanism, measured in one.
11Reproducibility
The measurement chain is replayable: the manifest schema (items as verified substrings of a fixed window, typed and source-attributed), the gist criterion, and the hop-tracking procedure are all mechanical given a session transcript. A two-arm comparator (vanilla compaction prompt versus typed extraction, single-hop and multi-hop at k=3 and k=6, scored by a calibrated semantic matcher rather than this notebook's lexical proxy, with per-arm token and dollar accounting) was pre-registered in full before its first run, including the hypothesis that our own arm fails on real prose for registered reasons. Its results replace the proxy figures here regardless of which way they fall; the number that ships will be the measured one.