How to Decide: Loop Transformer Blocks or Add More Layers

TL;DR Whether to loop transformer blocks or add more layers comes down to what’s actually scarce: a looped transformer reuses the same block weights across multiple passes — Nanbeige4.2-3B runs 22 blocks twice for 44 total applications from 22 weight sets — which halves parameters and, per the Mixture-of-Recursions paper, needs 6.8-18% less training compute to hit the same loss as an equivalent deeper model. It does not cut forward-pass compute or KV cache memory, both of which track total block applications, not weight count, so loop when parameters are your constraint and add layers when raw compute is.

Looped transformers answer a narrow but real question: what if a model didn’t need a distinct set of weights for every layer of depth? Reuse the same weights across two or more passes and you get more effective depth per parameter — which is exactly the trick a wave of 2026 models, including Nanbeige4.2-3B and the model reportedly behind GPT-6 Astra, are using. The idea isn’t new — Universal Transformers proposed depth-wise recursion back in 2018 — but it’s back because the parameter-versus-compute math finally has real numbers behind it, and those numbers cut a different way than most people assume.

If you’re deciding between a deeper model and a looped one, the mistake is treating this as free efficiency. It isn’t. This post is the decision math: what looping actually saves, what it doesn’t touch, and the procedure for picking the right one for your training budget.

What is a looped transformer, and why reuse blocks instead of stacking them?

A standard decoder-only transformer stacks N distinct transformer blocks, each with its own weights, and runs a token through all N once. A looped transformer instead takes a smaller stack of M blocks and runs the token through that same stack multiple times — the second pass reuses the exact weights from the first.

Nanbeige4.2-3B is the concrete case: 22 transformer blocks, applied twice, for 44 effective block applications total, but only 22 distinct sets of weights to store and train. The immediate win is obvious — a checkpoint half the size of a 44-block model with (allegedly) comparable depth of computation.

The less obvious part — the part that determines whether this is a good trade for your model — is that “comparable depth of computation” is doing a lot of work in that sentence.

How does looping actually change the compute and memory math?

Three things move, and only one of them moves in your favor:

• Parameters halve. 22 weight sets instead of 44 is a real, unambiguous win for checkpoint size, sharding, and anything gated on model size on disk.

• Forward-pass compute does not shrink. Every loop pass is a full pass through the blocks. Compared to running those 22 blocks only once, looping them twice “adds substantial work” — the model does roughly the same total FLOPs per token as a 44-block model without loops, not the FLOPs of a 22-block model.

• KV cache does not shrink either, and can’t be safely shared. Each pass needs its own cache, because the keys and values a token produces in pass one encode a shallower representation than the same token produces in pass two. Nanbeige’s team tried sharing one cache across passes to save that memory, and it degraded output quality — the two passes aren’t interchangeable, so collapsing their caches throws away real information.

What does move in your favor, per the Mixture-of-Recursions paper, is total training compute to reach a given loss: looped variants needed 6.8-18% less than a plain deeper model matched for parameter count and eventual quality. That’s a training-time win, not an inference-time one — worth being precise about, because it’s the opposite of what “compute-efficient architecture” usually implies.

Metric	22 blocks, no loop	22 blocks, looped ×2	44 distinct blocks
Distinct weight sets	22	22	44
Block applications per token	22	44	44
KV cache footprint (relative)	1×	~2× (separate per pass)	~2×
Training compute to match the 44-block loss	worse loss at equal compute	-6.8% to -18% vs. the 44-block baseline	baseline

Read the middle column against both neighbors: looping buys you the 44-block model’s quality and roughly its inference cost, using half its parameters and somewhat less total training compute to get there. It does not buy you the 22-block model’s cheap inference — that comparison is the one people skip, and it’s the one that decides whether looping is the right call for a latency-sensitive serving path.

If you’re already tracking KV cache memory as the actual serving bottleneck rather than parameter count, this table is the reminder that looping doesn’t touch that number at all.

Fixed, adaptive, and latent-reasoning loops: three variants, one choice

Three shapes of this idea are shipping right now, and they solve different problems:

1. Fixed looping (Nanbeige’s approach): every token gets the same number of passes, decided at training time. Nanbeige found two passes optimal for its compute-accuracy trade — more passes kept adding compute without proportionate quality gains.

2. Adaptive looping (Universal Transformers, extended by Mixture-of-Recursions): a lightweight router decides, per token, how many passes it gets. Easy tokens exit early; hard tokens loop longer. This is strictly more compute-efficient than fixed looping when your input distribution has a real spread of difficulty, at the cost of a router to train and tune.

3. Latent-reasoning loops: each pass is fed both the previous pass’s output and the original block input, so the stack keeps access to the unrefined representation instead of only ever seeing an increasingly processed one. This is the variant most associated with claims about hidden multi-step reasoning happening inside the loop rather than in visible output tokens.

Pick fixed looping for simplicity when your workload is fairly uniform in difficulty. Pick adaptive looping when it isn’t — the router earns its keep exactly when spending equal compute on every token would waste it on the easy majority, the same instinct behind tuning inference flags instead of buying a bigger GPU rather than paying a flat cost everywhere.

How do you decide: loop transformer blocks or add more layers?

1. Name your actual constraint first. If it’s checkpoint size, sharding cost, or parameter count for licensing/deployment reasons, looping is a real lever. If it’s inference latency or serving throughput, looping doesn’t help — go tune inference-time flags instead.

2. Check where you sit on the scale-versus-budget curve. The Mixture-of-Recursions results favor looping at larger model scale paired with a smaller training budget; the advantage shrinks toward zero as training compute grows unconstrained. If you can afford to just train the deeper model to convergence, do that.

3. Budget KV cache memory as if you were serving the deeper model, not the shallow one. Looping does not reduce cache footprint — plan capacity against the full block-application count, not the weight count.

4. Decide fixed versus adaptive by your input distribution. Uniform difficulty → fixed looping. Wide spread of easy/hard inputs → adaptive, and budget separately for training the router.

5. Re-benchmark forward-pass latency before shipping, not just parameter count. A model that looks half the size on disk but takes the same wall-clock time per token is not the win a smaller checkpoint implies — measure the thing you actually care about.

6. Only commit to looping if step 1’s constraint was genuinely parameters, not compute. If both are tight, a deeper model with fewer, more efficient layers is usually the simpler engineering bet than adding a router and dealing with per-pass KV caches.

Does looping mean GPT-6 Astra is hiding its reasoning?

This is the claim that put looped transformers back in the news: reporting that GPT-6 Astra uses a looped architecture, paired with speculation that looping lets it obscure reasoning steps from the visible trace. Worth separating the parts that are established from the part that’s a leap.

Established: Astra reportedly scores 99.9% on ARC-AGI-3 versus GPT-5.6’s 7.8%, a genuinely large jump, and its visible reasoning traces are shorter than predecessor models’ at matched or better accuracy. OpenAI has also hidden raw reasoning traces from end users since o1 — that policy predates any looping report by years and isn’t evidence of anything new.

The leap: that shorter visible traces mean the architecture is concealing computation that would otherwise be shown. OpenAI’s chief scientist has stated the model’s computation-graph depth is within a factor of two of GPT-4’s, and a shorter trace is at least as well explained by the model making fewer mistakes per step as by anything being hidden.

For a second concrete example of a benchmark number moving without a hidden-computation explanation, an agent harness change alone took an ARC-AGI-3 score from 13.3% to 38.3% with zero model changes. Score jumps on this benchmark have a documented history of coming from scaffolding and evaluation setup, not architectural mystery — exactly the alternative explanation worth ruling out before reaching for the more dramatic one.

None of this is settled — it’s an open research question the Mixture-of-Recursions authors and others are still working through, including whether shorter traces from looped models stay faithful to what the model actually computed. But “looped architecture” and “hidden reasoning” are two separate claims, and only the first one currently has public evidence behind it.

FAQ

What is a looped transformer?

A looped transformer applies the same stack of transformer blocks to a token more than once instead of stacking distinct blocks with separate weights for each layer. Nanbeige4.2-3B, for example, runs 22 blocks, then runs the same 22 blocks again on the result — 44 total block applications from only 22 sets of weights.

Does looping reduce inference compute or serving cost?

No. Every loop pass still runs a full forward pass through the blocks, so a 22-block model looped twice does roughly the same forward-pass work as a 44-block model without loops. The saving is in parameter count and checkpoint size, not in FLOPs per token or serving latency.

Why did sharing one KV cache across loop passes fail?

Nanbeige’s team tried reusing a single KV cache across both passes to save memory, and it degraded model quality. Each pass needs its own cache because the keys and values produced in pass one encode a different, less-refined representation than the same tokens produce in pass two — collapsing them into one cache throws away that distinction.

When does looping actually beat training a deeper model?

The Mixture-of-Recursions paper found looped variants form a better accuracy-per-parameter frontier at larger model scales paired with smaller training budgets; the advantage narrows and can disappear at maximum compute budgets. If you’re compute-unconstrained, a plain deeper model is the simpler choice.

Fixed looping or adaptive looping — which should I pick?

Pick fixed looping (a constant number of passes for every token) when you want the simplicity of Nanbeige’s approach and your workload doesn’t have a wide spread of easy versus hard tokens. Pick adaptive looping — a router deciding depth per token, as in Universal Transformers and Mixture-of-Recursions — when your inputs vary enough in difficulty that spending equal compute on every token wastes it on the easy majority.

Does GPT-6 Astra’s reported use of looped transformers mean it’s hiding its reasoning?

There’s no public evidence for that. OpenAI has kept raw reasoning traces hidden from users since o1, well before any looping report, and OpenAI’s chief scientist has said the model’s computation-graph depth is within a factor of two of GPT-4’s. A shorter visible reasoning trace is also just as consistent with a model making fewer mistakes as with anything being concealed.

If you’re weighing model architecture decisions more broadly, the same “measure the thing you actually pay for” discipline shows up in Rust’s dyn Trait versus generics memory cost and in when a 4B reinforcement-fine-tuned model beats GPT-5.6 — both are cases where the parameter or code-size number everyone quotes isn’t the number that actually determines the outcome. More on model and agent architecture trade-offs is in the LLM engineering topic hub.

Sources

• GPT-6 Astra, looped transformers, and hidden reasoning — Sebastian Raschka
• Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation — arXiv 2507.10524
• Universal Transformers — Dehghani et al., arXiv 1807.03819