Wink Pings

Deferred Tool Loading: Cutting Agent Costs by 62.5%

The bulk of an agent's cost isn't in the model; it's in the context. Baseten used deferred tool loading to cut the 141k-token overhead from 3 MCPs to 1.3k, reducing coding-task costs by an average of 62.5%. This overlooked optimization is becoming a must-have for agent architecture.

Agents are expensive, and model pricing is only part of it. Often, the money burns on unused tool definitions in the context window.

Baseten's Alex Ker recently published a field note about a practice called "deferred tool loading." The idea isn't complicated: don't stuff the entire tool list into the model all at once; load it when it needs it. He ran 8 benchmarks on Claude Code, and the results were direct:

  • Coding benchmarks (Terminal-Bench, SWE-bench Verified, SWE-bench Pro v2, DeepSWE v1.1): average cost down 62.5%, time reduced by 26.2%
  • Agent benchmarks (MCP-Atlas, AutomationBench, ComplexMCP, Toolathlon-GYM): average cost down 51.1%, time reduced by 14.9%

Why does it save so much? The problem is "eager loading."

Mainstream agent frameworks today, such as Claude Code, put every tool schema from every MCP server you register (Slack, Notion, Baseten, etc.) into the system prompt on every request. Each schema is a set of invocation instructions, and the model doesn't need all of them. Baseten measured it: with 3 MCP servers, tool definitions alone accounted for 141k tokens per request. And a model may take dozens of requests to complete a task, with every token being read repeatedly by the attention mechanism. Of those 141k tokens, most are tool definitions that will never be used, becoming a toll paid at every decoding step.

Deferred loading works like this: the system prompt contains only tool names and a small tool searcher (about 1.3k tokens). When the model needs a particular tool, it first calls the searcher to find the corresponding tool, and then the server inserts the full schema at the end of the conversation context. This way, the earlier KV cache is unaffected, and the amount of content each output token needs to "look back" at is greatly reduced. Think of it like lazy loading in frontend: resources are loaded only when they are actually needed.

Baseten implemented this capability on APIs for open models (DeepSeek, Kimi, GLM), supporting both the Chat Completions and Messages API formats. That means when using third-party open models, you can also get the same deferred tool loading as with the OpenAI/Anthropic APIs. Note that when Claude Code and Codex are pointed at third-party model providers, they often turn this option off; you need to enable it manually, and the other side's gateway must support it.

Community feedback also confirms the idea. Viv said under that tweet: "The alpha in harness engineering is getting context engineering delivery right." She calls the LLM context window "the sacred boundary where all computation happens"; protect it and fill it with the right things, and agents will perform better. Daniel Priscu put it more bluntly: keeping junk out of the window is probably 90% of the work.

Of course, this isn't a panacea. In the benchmarks, Toolathlon-GYM tasks required an average of 68 tool calls, and eager loading was actually faster; DeepSWE and MCP-Atlas were also slightly slower, but the cost savings were enough to offset it. In addition, Loay pointed out that simply removing unused MCP servers in Claude Code can also recover that 140k-token waste, without switching gateways.

Sid Potti made a 6-minute explainer video with Armature that visualizes the process very clearly:

Now, open your Claude Code, run /mcp, and see which tools you've never used. Delete them first, then talk about optimization.

发布时间: 2026-11-20 16:14