Same Bug Through Round 28: Who Among 30 AI Agent Products Lost the Critical Port?

Round 3: the coding agent runs tests, and the log has a line: connection refused, db-primary, port 5433. Round 28: it needs to write a fix and needs this port. The 24 rounds in between read files, grep, edit code, and run small tests. Round 17: the context management mechanism triggers.

Sergii Makarevych replayed the same 30-round coding ticket. Ticket #4821: integration tests report a database error in CI. 8 strategies, more than 30 products; model calls use Claude Haiku 5.5; token counts use the o200k tokenizer; bills are normalized to Claude Sonnet 5.5 public pricing: $2 per million input tokens, one-tenth for cache reads, output not counted. The numbers describe only this ticket and do not represent your workload.

The baseline is Append everything, deleting nothing. The 30-round input bill is $0.77, peak window 156,635 tokens, cache hit rate 93.5%. It is the cheapest because it never changes the prefix, so almost every token is a cache read. It is also the most dangerous because by round 28 the window size already exceeds the range where LOCA-bench measured accuracy starting to drop. On this ticket, the port is still in the window, and the model uses it.

Results for the 8 strategies:

The most striking number is truncation. It saves only 2% but loses the port along with old messages. Clearing saves more, about one-fifth, and also loses the port. The reason is that the test log is part of the cleared tool results, and the model's round 4 reply did not repeat the number. Summary compaction preserves the fact, but the summary call reads the entire window at uncached price, so it saves only about one-tenth. Summary with instructions preserves the error block and pytest statistics verbatim; the form is better, and the bill barely changes.

trim and offload are the cheapest, with bills only one-fifth of the baseline. They preserve the port because of position and storage. trim keeps the head and tail of the log, and the error line happens to be at the start. offload writes large results to files, leaving only a preview and path in the window. In round 28 the model opens three offloaded files but not the right one. The fact is recoverable; the agent does not recover it. The distinction matters. The mechanism determines whether the fact is present; the model determines whether it is used.

isolation puts the test run into a sub-agent, and the main window gets back only a 2,000-token report. The port is in the report, and the model uses it. But five sub-agent calls add 52,573 tokens, and the main window still grows to 109,416 because file reads are not delegated. Sub-agents suit tasks with large output and short conclusions, such as running tests, searching the web, or scanning directories. They are not suitable for editing a file that the parent agent must then understand.

Sergii's conclusion is not complicated. Most products by default clear or compact at around 100,000 tokens, which is the most reliable mechanism for removing early facts. Compaction preserves facts only when the summary prompt explicitly requires preserving exact values, and the summary call costs roughly one uncached full-window read. Cheap strategies change what enters the window, and the agent must actively retrieve what was moved away.

The market map has four layers. Model APIs do compaction and clearing, with the fewest actions and the least control. The Anthropic Messages API has server-side compaction, tool-result clearing, thinking clearing, a client-side memory tool, and lazily loaded tool search. The OpenAI Responses API has threshold compaction and a compaction endpoint; outputs are encrypted items that outside systems cannot read. xAI has a developer-triggered compaction endpoint, with no automatic threshold. Gemini has no compaction; its answer is a one-million-token window and caching. Framework middleware gives control, but defaults are minimal. Deep Agents summarizes at 85% of the window, offloads tool results over 20,000 tokens to a virtual filesystem, and isolates sub-agents. Letta compacts when history no longer fits, summarizes 30% of the sliding window, and also has searchable evicted messages and pinned memory blocks. Mastra at 30,000 tokens has an observer agent turn raw messages into notes. Coding harnesses have the most defaults and the least documentation. Claude Code compacts at the model limit, clears old tool results, removes old images in batches, and nests sub-agents three levels deep. Codex CLI compacts at 90% of the window, then keeps only user messages up to 20,000 tokens plus a summary. Gemini CLI compacts at half the window, and truncates tool output over 40,000 characters and writes it to a file. Cursor summarizes near the top of the active window, and moves long shell and MCP output to files. Standalone compressors and gateways are the only place where you can compress tokens. LiteLLM polyfills Anthropic's clearing edits on every provider, configures a summary model to do compaction, and can call Compresr or Headroom as a guardrail. Headroom processes tool results with a JSON crusher, code compressor, or learned compressor, keeps the original text by hash, and gives the model a retrieve tool.

Caching is the part most product pages do not mention. Each provider discounts only byte-identical prefixes. The agent loop is an ideal customer: every round appends to the same prefix. As soon as a context mechanism edits history, the prefix is invalidated, and content after the edit point must be written to cache again. On Sergii's ticket, clearing lowered cache hit rate from 93.5% to 88.2%, and compaction to 86.9%, because the round-17 edit rewrote a prefix once. trim and offload keep 93%, because they modify only when the result enters the window and never touch it again. isolation keeps the main window at 93.4%, with each sub-agent opening its own cache. Manus goes further: it bans timestamps and non-deterministic serialization, never edits previous actions or observations, only masks tool availability, and does not delete tool definitions. The ACON paper reaches the same conclusion from the cost side: compacting history rarely saves money because it breaks the cache; compacting each observation once when it arrives is more cost-effective.

In the replies, someone asks whether compaction fabricates details or only checks that key facts survive. Someone asks whether, if the fact needed in round 28 was corrected midway, the summary would keep the old value and lose the correction. Someone asks whether compaction preserves the reasons for decisions, not just the numbers. Another asks: if clearing tool results works fine for 27 rounds and then silently fails in round 28, which is the worst failure mode, will the product tell you in advance what it threw away? Sergii does not expand on these in the tweet replies. The SelfCompact study he cites offers a warning: when summaries change answers, about 40% of the changes are from right to wrong; fixed-interval summarization scores lower than letting the model choose when to summarize. LOCA-bench's numbers are more direct: without context management, the strongest model's accuracy in an 8,000-token environment is 96.0%, drops to 65.3% at 64,000 tokens, 34.0% at 128,000, and 14.7% at 256,000. Clearing tool results improves accuracy by only 1.3 to 2.7 percentage points at the 128,000-token level. Programmatic tool calling raises one model from 38.7 to 49.3.

In the replies, another developer posts Kora, an open-source app for Apple Silicon Macs that orchestrates Claude Code, Codex, OpenCode, and Antigravity into a lead plus workers, using a local code graph to stuff relevant declarations into context before the agent starts searching. Its approach complements Sergii's test: Sergii compares how to prune after reading, while Kora wants to reduce what the agent has to read in the first place. Kora's workers each run in their own git worktree, and before merging it uses the code graph to check changes that git can merge cleanly but that would break each other. It creates no account, has no telemetry, and is MIT-licensed.

If you want one practical shape to take away from this material, Sergii's advice is: do a size trim once at the entrance, give facts a fixed storage place, use a threshold mechanism as a fallback, and hand sub-tasks with long output and short conclusions to sub-agents. Every stack has these four things; products differ in which ones they give you.

A more specific order for checking:

First find the decision points. Model APIs, framework middleware, coding harnesses, and agent gateways: each layer can edit only the window it can see. A gateway cannot see a harness's sub-agents.

One mechanism manages size, another manages facts. Use trim, offload, or compaction for size. Use pinned blocks, memory files, project rule files, or sub-agent reports for facts. Do not let summarization carry fact preservation alone.

Lower the default trigger threshold. A default clear at 100,000 tokens, or compaction at 90% of the window, often acts only after accuracy has already started to decline. LOCA-bench's degradation begins at around one-third of the window.

Write summary instructions clearly, requiring verbatim preservation of error lines, file paths, and numbers. Sergii's default summary marked the port as a possible port mismatch, unverified. A custom instruction leaves the error block and test statistics intact.

Clear in batches to preserve the cache prefix. Measure cache hit rate and look at it alongside the percentage of tokens saved. For product pages that report only token savings and not hit rate, discount the numbers.

Context management should first answer: on which day, in which round, which fact will be thrown away, and do you have a place to pick it back up? How many tokens are saved comes later.