Garry Tan Deleted a Skill, and the Comments Started Doing the Cost Math

Garry Tan posted a tweet saying he deleted the /investigate agent skill. The reason was simple: Opus 5.5 does better without it.

He also offered a bigger judgment: it is necessary to build an eval-driven development loop for agent skills. Each new model generation is stronger, and you can delete things you no longer need. The original tweet link is here.

The statement isn't complicated, but the comments break the problem down more finely.

First, some background. An agent skill can be understood as a fixed bundle of instructions and procedures for an agent, like a work manual. When the model isn't capable enough, the manual provides a fallback. After the model gets stronger, the manual may become a ceiling, or it may become a cost tax paid on every run.

So when Garry Tan deleted /investigate, on the surface he was cleaning up one skill, but in practice he was running a subtraction test after a model upgrade.

In the comments, Alpha Batcher asked more sharply: what's really interesting is how the eval loop makes the deletion decision, and how many of the other skills can withstand a head-to-head comparison with the bare model.

John Rood pointed out an easily overlooked point. Deleting a skill is easy, but the skill may still contain exclusions added after earlier pitfalls. The model catching up does not mean those exclusions automatically become invalid. You need to check what it was originally preventing, not just whether it still helps now.

Jeff Bruchado added the second half: after deleting the skill, the /investigate eval should keep running. If some future model regresses in this area, this eval is the signal to bring it back.

Tavio offered an actionable method: pick a skill; first write down clearly what it was originally meant to accomplish, which failures it prevented, and under what conditions it is most valuable; using the same model, same tools, and same budget, run three comparison groups in fresh sessions—full skill, simplified skill, and no skill; record successful outcomes, cost per accepted result, execution time, and regressions; finally decide whether to keep, shorten, constrain, or retire it. If there isn't enough evidence, write down the next experiment clearly.

People are also doing the cost math. Someone in the comments mentioned that fixing a bug costs about 1.4x, 43% more, or 38 cents more per bug. These numbers come from the reply discussion, not the original tweet text. But they point to a reality: a skill can be neutral for accuracy and still impose a tax on every run.

One reply put it more bluntly: most skills are just notes written for a model that has already graduated. Someone else joked that the most reliable finding from /investigate was that each bug cost 38 cents more.

Behind these complaints is a serious judgment. With every model release, the skill list should have a chance to shrink. Only growing and never shrinking is like continuously wrapping a moving target. As Sachin Vedaraj put it, if the skill list only grows, you're just collecting wrappers.

But subtraction doesn't mean deleting at random. Vincent Koc said you can't vibe an eval. Jatin Garg also reminded people that you can't conclude from one test that the bare model is better at fixing bugs, but you can delete a skill because of the extra cost. Nash mentioned that early Claude Code Opus needed strong constraints because it would really rm -rf the entire repo; now many Claude.md files are more like vanity configuration. The comparison is concrete: constraints used to be a seatbelt, and later they may become a straitjacket.

So what's worth watching in this tweet isn't 'another skill got deleted.' It's whether, after a model upgrade, the team has a loop to ask: what problem is this skill still solving, have the failures it prevents been recorded, who will catch regressions after it's deleted, and is anyone doing the cost math.

Skills are not medals. The stronger the model, the less scaffolding you need. Before deleting, write down what it was guarding against; after deleting, keep testing. Phrases like 'AGI tide rising' can serve as background music; what actually needs doing is still the boring comparative benchmarking and regression cases.