16 Tokens as Memory: Stanford Open-Sources Memorilla, Cutting Document Token Usage by 108x for Frozen LLMs

Stanford researchers have launched Memorilla, an open-source memory system. Asif Razzaq from MarkTechPost laid out several core metrics in his coverage: the system compresses batches of documents into only 16 memory tokens to feed into a frozen Qwen3-8B model, it reaches 84.33% performance on FactKG, and its document token usage is 108 times lower than that of the full context setup.

Three key points here are worth breaking down separately.

Frozen Model Design. The weights of Qwen3-8B stay completely unchanged, and Memorilla operates as a pluggable external memory module. The original paper refers to it as a latent memory module built specifically for frozen LLMs. For developers, this means memory capabilities can be iterated independently, with no need to retrain the base large language model every time you update the memory functionality.

16 memory tokens. The standard full-context solution feeds raw full documents directly into the model's context window, while Memorilla first compresses the entire document collection into an extremely small set of tokens. The 108x reduction gap comes entirely from this compression. The tradeoff is also very clear: how much detailed information 16 tokens can retain depends on how the compression module is trained, and whether your task requires verbatim references to the original source material. The public results only include performance on FactKG, so do not directly extend this conclusion to all general use cases.

84.33%. This is the benchmark score on FactKG, and it only carries meaningful weight when considered together with the "108x fewer document tokens than full context" metric. The accuracy figure alone does not give the full picture, nor does the compression rate in isolation.

Memorilla is currently an open source project. The original coverage does not elaborate on the code repository, training details or performance on other benchmarks, so users who want to reproduce its results or integrate the system will need to follow up on the official project page for more information. At the very least, this research confirms a clear promising direction: apart from expanding long context windows, there is another viable path that compresses documents into a small set of latent memory tokens for frozen base models to use. Whether it fits your use case depends on your task's requirements for fine-grained detail fidelity.

Source: Stanford Research Open-Sources Memorilla, a Latent Memory Module for Frozen LLMs, written by Asif Razzaq.