Turning Visual Memory into a Lookup Table: V-LUMEN Opens Another Capacity Axis for VLMs

V-LUMEN frames the capacity problem of vision-language models as a lookup problem.

Language models already have a capacity-scaling idea: use lookup-based memory to add capacity without making the model deeper, making it wider, or piling on dense compute. Moving this to vision-language models gets stuck because visual features are continuous, have spatial structure, and are query-dependent; unlike discrete text tokens, they do not naturally support n-gram lookup.

The paper's approach is to use a frozen codebook to discretize visual features, build memory keys carrying spatial information through spatially aggregated hashing, and then modulate them with text-conditioned relevance signals. All addressing is non-parametric, and the retrieved embeddings are fused into the backbone through a gated residual path.

On results, the paper claims clear improvements on a frozen Qwen3-VL-2B backbone across five reasoning-intensive benchmarks, and zero-shot transfer to three benchmarks whose images are disjoint from the training data. There is another detail more worth examining than the score gains: when matching the parameter budget, moving part of the capacity of a LoRA or MoE adapter to V-LUMEN keeps accuracy on both backbones. The authors accordingly say external visual memory is an independent capacity axis, rather than packaging it as a stronger adaptation method.

The cost structure is also worth noting. Lookup cost is constant with respect to table size and occurs only once during prefill. Capacity growth lands in storage, not in per-token compute. For those doing inference deployment, this is more worth accounting for than yet another adapter: memory and adapters can be configured separately.

The paper has been accepted to NeurIPS 2026; code is at GitHub, and the paper is at arXiv.