Looping Transformers 32 Times: Meta Finds Returns Hit a Ceiling and Then Reverse

Meta FAIR and MIT have released a paper dedicated to one question: how should a Transformer loop so that compute isn’t wasted?

The paper is titled How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference, arXiv ID 2610.40325. The core approach is not new: reuse the same set of weights and repeatedly apply the same Transformer, or part of it, to token representations, making the model deeper without increasing parameter count.

What is really worth looking at is the experimental setup. The authors train dense decoder-only models from scratch, up to 1B parameters, up to 155B training tokens, up to 1.07×10^21 FLOPs, and as many as 32 loop iterations. This scale is not frontier large-model scale, but it is enough to isolate several engineering constraints.

Here are the conclusions up front.

With a fixed parameter budget, looping the entire Transformer usually gives the strongest quality. When training FLOPs are constrained, looping a middle segment of blocks gives better returns. More loop iterations are not always better either; gains saturate, reverse, and can even become unstable. The paper puts it plainly: adding loops is not consistently beneficial.

This also explains why some looping papers favor the full stack while others only touch the middle layers. The answer changes depending on whether parameters or compute is the binding constraint first.

Another detail is adapters. The authors compare strict weight sharing with adding a small low-rank adapter to each loop, so that each iteration learns a slightly different transformation. When there are few loop iterations, these adapters provide limited benefit; once weights are reused more times, the adapters’ effect becomes stable.

The parallel inference part is more interesting. Ordinary looping slows down decoding, with latency growing roughly with the number of loops. Parallel Looped Transformers try to pass hidden states between adjacent token positions so that the looping stages can run concurrently. But the authors find that, in their experiments, the original formulation does not reliably preserve the quality gains of ordinary looping, and quality can degrade when more loops are added.

Their fix is not flashy: keep the current token representation, normalize it and the loop state separately, and then control how they are combined. After this change, PLT recovers strong performance across different model sizes and training budgets.

The questions in the comments are closer to engineering practice than the abstract. Some ask whether extra loops help more on reasoning-intensive tasks or are broadly effective; others ask when 32 loops at the 1B-model scale hit diminishing returns. The tweet summary does not unpack these answers, so you have to go through the main text and appendix. The arXiv page shows the paper is 149 pages including appendices, so there is a lot of information.

Some also say that looping parameters recall earlier recurrent neural networks. That association is not surprising. Looped Transformers save parameters but spend training compute and inference latency. It is a kind of resource trade-off, not free depth.

Paper link: https://arxiv.org/abs/2610.40325