The open-source AI stack is becoming composable, but test tool calls before switching models

Open-source models are getting closer to production. Many teams want to switch over from closed-source and think it is just a matter of changing an API, only to find a whole stack standing behind it: models, inference, gateways, harnesses, and tools. This stack is now becoming composable.

In the blog post "The Open Source AI Stack", Together AI summarizes this stack as MIGHT. M is the model, responsible for understanding requests and deciding what to do; I is inference, providing the infrastructure on which the model actually runs; G is the gateway/router, deciding which model or provider a request is sent to; H is the harness, managing conversations, giving the model tools, and connecting to codebases; T is tools, including skills and MCP servers. The five layers are independent, and each can be replaced separately. If you want to switch models, you do not need to rewrite the workflow; if you want to switch providers, you do not need to touch the code. After a new model is released, you can try it in minutes.

The model layer is most people's first reaction. But bigger is not always better. Large models are stronger on ambiguous, complex tasks, such as refactoring an authentication system, upgrading a framework, or reviewing a PR. Small models are not far behind large models on well-defined tasks, and they are much faster and cheaper. The blog gives a comparison: Kimi K3 has 1.8T total parameters and 104B active parameters; GLM 5.3 Flash has 320B total parameters and 18B active parameters, roughly 6 times smaller and 20 times cheaper. For everyday work like changing a function or adding tests, a small model is enough. Treat models as tools, not as religion.

To discover new models, you can look at leaderboards such as The Open Frontier and Artificial Analysis, but do not treat leaderboard scores as the only criterion. Your real workload is what counts.

Models run on inference providers. The same model on different providers gives roughly the same results, but pricing, speed, and API features differ. Because not all providers host all models, the gateway and router layer becomes useful. OpenRouter and Vercel AI Gateway aggregate multiple providers behind one API; LiteLLM lets you set up your own router. Switching models becomes a matter of changing one line of configuration.

The harness is the program you run on your computer, managing conversations, giving the model tools, and connecting to the codebase. The model decides what to do, and the harness is responsible for execution. So the quality of a coding agent depends not only on the model, but also on whether the harness organizes context, applies patches, recovers from errors, and asks for permission. The same model can feel completely different in two harnesses. Common ones include PI, OpenCode, and Amp. If you already know how to use a coding harness, the barrier to switching to open-source models is lower than you might think. Closed-source harnesses such as Claude Code can also connect to open-source models through Together Link.

The tool layer consists of Skills and MCP. Skills are reusable instructions, and MCP is a standard protocol for connecting external tools. Both extend the harness's capabilities.

However, composability also brings new problems. One commenter put it well: keep a set of tool-calling regression cases before switching models. Being able to connect to the interface is only one hurdle; whether multi-turn state, error retries, and structured output can keep running is another. Composability does not mean painlessness; replacing each layer may introduce new behavioral differences.

Context management is also an easily overlooked point. The longer the session, the more the model has to reason about, and the worse the results. When switching models, tasks, or lines of thought, it is best to start a new session. The blog also recommends a workflow: a large model makes a plan, a small model implements it item by item, and then another large model reviews it. Divide the work among three models; do not let one model carry everything the whole time.

The most exciting thing about open-source models is that every component of the entire stack can be upgraded independently. The number of models is only the beginning. You no longer have to choose an AI coding product. You can assemble one yourself. This is not good news for closed-source vendors.

Content reference: Together AI blog "The Open Source AI Stack", author Hassan El Mghari. The tweet comes from @togethercompute. The commenter's view in the article comes from Marvin's reply.