Teaching Pretrained Language Models to Think Deeper with Retrofitted
  Recurrence
Recent advances in depth-recurrent language models show that recurrence can
decouple train-time compute and parameter count from test-time compute. In this
work, we study how to convert existing pretrained non-recurrent language models
into depth-recurrent models. We find that using a curriculum of recurrences to
increase the effective depth of the model over the course of training preserves
performance while reducing total computational cost. In our experiments, on
mathematics, we observe that converting pretrained models to recurrent ones
results in better performance at a given compute budget than simply
post-training the original non-recurrent language model.
691312e3a644ba07c499c89d
Sean McLeish
claimed_verified
2025-11-11T19:41:40.749Z
65255f1073a043e50d043641
/avatars/257085f01c439d7c84787a4e6d085b3d.svg
Sean McLeish
smcleish
user
smcleish
691312e3a644ba07c499c89e
Ang Li
claimed_verified
2025-11-11T19:41:37.955Z
64a1b18b98fad0c8a5b04e3a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8EvZSqz_GumooxK0uHkyR.png
Leon Li
leonli66
user
leonli66
691312e3a644ba07c499c89f
John Kirchenbauer
691312e3a644ba07c499c8a0
Dayal Singh Kalra
691312e3a644ba07c499c8a1
Brian R. Bartoldson
691312e3a644ba07c499c8a2
Bhavya Kailkhura
691312e3a644ba07c499c8a3
Avi Schwarzschild
691312e3a644ba07c499c8a4
Jonas Geiping
691312e3a644ba07c499c8a5
Tom Goldstein
691312e3a644ba07c499c8a6
Micah Goldblum
Teaching Pretrained Language Models to Think Deeper with Retrofitted
  Recurrence
Recent advances in depth-recurrent language models show that recurrence can
decouple train-time compute and parameter count from test-time compute. In this
work, we study how to convert existing pretrained non-recurrent language models
into depth-recurrent models. We find that using a curriculum of recurrences to
increase the effective depth of the model over the course of training preserves
performance while reducing total computational cost. In our experiments, on
mathematics, we observe that converting pretrained models to recurrent ones
results in better performance at a given compute budget than simply
post-training the original non-recurrent language model.
Converting pretrained non-recurrent language models to depth-recurrent models improves performance at a given compute budget using a curriculum of recurrences.
depth-recurrent language models recurrence curriculum of recurrences effective depth computational cost mathematics post-training