Six Fixes in LLM Architecture: From Forgetting Words to Thinking

In the Outcome School blog, Amit Shekhar breaks the evolution of LLM architecture into six stages. The throughline is simple: wherever the previous version hurt, the next version patches it. The evolution of large-model architecture looks more like bug fixing than god-building.

First, let's fill in the context. LLM architecture is the blueprint for how a model reads text, remembers information, and produces the next word. If we unpack LLM, Large means lots of data and lots of internal settings, Language means processing text, and Model means learning patterns from large amounts of text. Architecture determines how all of this is organized.

Stage One: RNN reads one word and updates a little note

RNN reads a sentence from left to right, and after reading each word it rewrites the same small memory. Short sentences work; long sentences expose the problem. By the time it reaches the end of the sentence, information from the beginning has been overwritten repeatedly. Riya appears at the beginning; by the time it reads she, the model has a hard time confirming who she refers to. Another problem is speed. Word 1 must wait for word 2, so training cannot be parallelized. LSTM and GRU alleviated forgetting but did not cure it.

Stage Two: Attention spreads the whole page open

Attention lets the current word look directly back at every previous word and then decide which ones matter. When processing she, Riya might get a weight of 0.70, went 0.10, market 0.15, and then 0.05. The weights add up to 1, and the model mixes information from these words according to the weights.

There are three things here. Query is the current question, Key is the label for each word, and Value is the actual content inside the word. The model uses Query to match against Keys; words with higher matches contribute more Value. Self-Attention is when words within the same sentence look at each other. Multi-Head Attention is several attention groups running in parallel: one group watches names, one watches grammar, and one watches position.

Attention solved forgetting, but early on it was often added on top of RNN. RNN was still reading word by word, so the speed problem did not disappear.

Stage Three: Transformer removes RNN

The 2017 paper 'Attention Is All You Need' asked a direct question: since Attention is doing the main work, why keep RNN? Transformer removes RNN and stacks attention plus feedforward networks into blocks. All input words can be processed in parallel, which matches GPU parallel computation. RNN processing 6 words takes 6 steps; Transformer can process them at once.

Reading the input in parallel does not mean writing the answer in parallel. During generation it still emits one word at a time, adds the new word back to the input, and then predicts the next one. This is autoregressive generation. Reading is parallel, writing is serial; this distinction is key later in KV Cache.

Transformer also had to solve the word-order problem. All words come in together, and the model does not know who comes first or last. Positional encoding adds a position signal to each word. Early on it used fixed signals; later models such as LLaMA used RoPE. The method changed, but the purpose did not.

Stage Four: Scaling makes the model bigger

Scaling means adding parameters, data, and compute at the same time. Around 2020, researchers found that adding all three together smoothly improves model quality. GPT-2 had about 1.5 billion parameters, GPT-3 about 175 billion, more than 100 times larger a year later. GPT-3 could write articles, answer questions, and write code, and these abilities were not specifically trained.

The 2022 Chinchilla paper added a correction: many large models had too many parameters and too little data. A smaller model with more data might beat a larger model. Later LLaMA trained on trillions of tokens, following this path.

Stage Five: MoE makes large models cheaper

In a dense Transformer, every word goes through all parameters. If a model has 175 billion parameters and you ask 100 words, every word makes all 175 billion parameters work. Double the model, and you double the cost.

MoE splits the feedforward network into multiple experts and adds a router. Each word selects only a few experts, usually top 1 or top 2. It is like hospital triage: a patient does not get examined by every department at once.

Mixtral 8x7B is often misread as 56B. It actually has about 47B total parameters, because only the feedforward networks were copied; attention and shared parts were not. Each word activates about 13B parameters. DeepSeek-V3 is more extreme, with about 671B total parameters and about 37B activated per word.

MoE saves compute, not memory. All experts have to be loaded, because the next word may switch experts. The router also needs to do load balancing; otherwise all the work gets pushed onto a few experts, and the other experts learn nothing.

Stage Six: Long context, Mamba, multimodal, inference-time scaling

Attention has to compare against every word. 1,000 words means 1 million comparisons; 100,000 words means 10 billion comparisons. Double the input, and the cost quadruples. KV Cache also has to store historical Keys and Values, so long context eats up a large amount of memory.

GQA lets multiple attention heads share Key and Value, and LLaMA 3 uses it. MLA compresses Key and Value before storing them, and DeepSeek uses it. Sliding Window Attention makes each word look only at a nearby window, and Mistral uses it. All three methods are trying to reduce the cost of long context.

Mamba takes another path. It keeps a running memory like RNN, its cost grows linearly with input length, and it decides which information to keep and which to drop. Hybrid models combine a few Attention blocks with many Mamba blocks, aiming for Attention's precision and Mamba's speed.

Multimodal models cut images into patches and treat them as tokens. Audio and video work similarly. Reasoning models spend compute at answer time: first writing a long chain of thought steps, then giving the answer. Scaling now has two lines: training-time scaling and inference-time scaling.

The original article also includes a video whose theme is the basic puzzle pieces of LLM, RAG, MCP, Agent, Fine-tuning, and Quantization: AI Engineering Explained.

A table to see the six stages clearly

Looking down this line, each stage keeps the useful parts of the previous stage and then fixes the most painful problem. Next time you see a new model name, first ask which pit it patches; that is more useful than just looking at the parameter table. The original article comes from Amit Shekhar's blog: Evolution of LLM Architecture.