The Token You Throw Away: Full-Bandwidth Transformers Recycle Hidden States

Every autoregressive transformer runs two feedback loops: horizontal, across the sequence of generated tokens, and vertical, through the layers of the stack. The horizontal loop is well-studied. The vertical one has a quiet flaw: after each decoding step, the top-layer hidden state, which encodes everything the model computed about the current context, gets discarded. Only the sampled token comes back. That token is a compressed, discretized shadow of the computation that produced it.

This is not a minor inefficiency. It is a structural bottleneck. The model spends billions of parameters building a rich internal representation, collapses it to a single vocabulary index, and then forces the next step to reconstruct context from that index alone. The depth budget resets. The non-verbalized computation, everything that didn't make it into the chosen word, is gone.

The full-bandwidth transformer fixes this with one design change: latent feedback. At each decoding step, the previous top-layer hidden state is fused with the sampled token embedding through a gated linear unit, and the result feeds back as the next input. The hidden state re-enters the stack with a full depth budget to process it again. The standard architecture, KV cache, and language-modeling objective stay intact. Nothing else changes structurally.

The training challenge is real: standard teacher forcing runs all positions in parallel, but latent feedback creates a sequential dependency between steps. The solution is a scheduled multi-pass objective. Feedback is introduced late in pretraining, and a small fraction of deeper feedback passes are mixed in for stability. The model learns to use the recycled signal without losing the efficiency of parallel training.

At 1B parameters trained to 400B tokens, latent feedback improves validation loss, 5-shot language-model benchmarks, math and coding generation, and instruction-tuned performance. The per-token decoding overhead is negligible. Full-bandwidth transformers match or approach standard transformers trained on roughly 1.5 times more tokens. On reasoning tasks specifically, they produce shorter traces at equal or better accuracy, meaning the recycled hidden state lets the model do more computation per token rather than more tokens per answer. For teams training or fine-tuning models where compute budget is fixed but token budget is not, the takeaway is direct: this architectural change is a free multiplier on effective training data.

We're thinking: We find the reasoning-trace result more telling than the validation loss improvement. Shorter traces at equal accuracy means the model is compressing deliberation into its hidden-state channel rather than externalizing it as tokens. That has a direct cost implication: if chain-of-thought length is a latency and cost driver in production, a model that reasons more densely per step is worth more than its benchmark score suggests. The open question is whether latent feedback interacts well with speculative decoding or other inference-time acceleration methods, since the sequential dependency introduced by the gated feedback unit may complicate draft-verification pipelines. That is the caveat teams should pressure-test before committing to the architecture at scale.

Key takeaways:

• Latent feedback fuses the top-layer hidden state back into the next input via a gated linear unit, giving non-verbalized computation a renewed pass through the full model depth without changing the base architecture or KV cache.
• A 1B-parameter full-bandwidth transformer trained to 400B tokens matches or approaches a standard transformer trained on roughly 1.5× more tokens, with negligible per-token decoding overhead; the multi-pass training schedule is the primary caveat for teams adapting this to existing pretraining pipelines.
• Teams training or fine-tuning models under a fixed compute budget should evaluate this architecture as a token-efficiency multiplier, particularly for math and coding tasks where reasoning trace length directly affects inference cost.

Source: Full-bandwidth transformer