AI Cost Optimization: Don't Just Swap in Smaller Models—Among Six Levers, Start with Caching and Routing

Alex Xu listed six cost optimization strategies for AI systems, with an infographic and a short video. The original tweet is here. The video puts all six methods in one diagram, which is a good way to build a big-picture view first.

The six are:

1. Prompt Caching: Cache stable prompt prefixes to avoid repeated computation. System prompts, tool definitions, and fixed long documents are all good candidates to put at the front.
2. Model Routing: Route to the most economical model based on query difficulty. Don't send simple questions to the most expensive model; escalate only difficult questions.
3. Batch API: Process large numbers of requests in batches, trading asynchrony and latency for cost. Suitable for offline tasks, data cleaning, and batch labeling.
4. Semantic Cache: Reuse answers to similar questions without calling the model again. The higher the hit rate, the more you save, but you need to handle answer expiration and consistency issues.
5. Trim the Context: Limit history and retrieve only the content currently needed. The longer the context, the more both cost and noise increase.
6. Go Smaller: Create smaller, streamlined, or quantized models for specific tasks. When the domain is fixed and the task is narrow, a small model is often sufficient.

The ordering of these strategies matters. Caching-type strategies usually come first because they require small changes and deliver direct benefits. Routing is the lever most often pulled in real systems; as replies have noted, hand routine work to cheaper models. Batch API saves money at the cost of waiting time, making it suitable for scenarios where results are not urgently needed. Semantic Cache looks great, but in practice you need to watch the hit rate and the reuse of wrong answers. Trim the Context is often underestimated; it not only reduces tokens but also reduces the chance of the model being led astray by irrelevant information. Go Smaller is more stable later on: first confirm task boundaries and an evaluation set, then talk about distillation, quantization, or specialized small models.

A counterintuitive point is that cost reduction often shifts the bill from model fees to engineering complexity. Caches need invalidation maintenance, routing needs trained classifiers or written rules, and semantic caches need similarity threshold evaluation. Focusing only on model unit price makes it easy to miss these long-term costs.

If you look at only one metric, first calculate the average cost per request, then break it down into model, tokens, and cache hit rate. The bill won't automatically look better just because you switched to a smaller model.