Large Models Don't Read Words—They Only Recognize Tokens
Why do large models speak in tokens? Ben Dicken explains it clearly in a video: LLMs take in embeddings, humans speak in words, and tokens are the intermediate conversion layer. In the comments, someone compares it to an exchange rate, while another reminds: API bills are also charged by token.
Ben Dicken posted a video titled “Why does an LLM speak in tokens?” On the left side of the screen is a two-layer neural network, with red arrows marking forward propagation and white arrows marking backpropagation; on the right, Ben, dressed in black, explains: large models take in tokens, not words.
Why can't we feed words directly into the model? LLMs process embeddings, while humans speak in words. Between words and vectors, there is a token layer. Tokens cut sentences into the smallest units the model can digest, then map them into vectors.
Someone in the comments summed it up well. Guido said: tokens are an exchange rate—words are converted into tokens, tokens are then converted into embeddings, and in the end you are billed in a currency no one actually speaks. This analogy is clearer than many tutorials.
Another commenter asked whether the original Transformer paper “Attention Is All You Need” is the best way to understand the embedding step, and whether it is beginner-friendly. Ben replied that he has not finished reading it yet and wants to do a livestream later.
Readers with a development background will pay more attention to Sunil's addition: the server side is also billed by token. Context limits, KV cache memory, and throughput are all tied to tokens. For the same piece of text, the token counts produced by different models' tokenizers can differ quite a bit, which can be surprising the first time you compare costs.
Next time you see the token count on an API bill, you will probably have a sense of what is happening behind it.
发布时间: 2026-10-26 12:19