Progressive Residual Warmup for Language Model Pretraining
Transformer architectures serve as the backbone for most modern Large Language Models, therefore their pretraining stability and convergence speed are of central concern. Motivated by the logical dependency of sequentially stacked layers, we propose Progressive Residual Warmup (ProRes) for language model pretraining. ProRes implements an "early layer learns first" philosophy by multiplying each layer's residual with a scalar that gradually warms up from 0 to 1, with deeper layers taking longer warmup steps. In this way, deeper layers wait for early layers to settle into a more stable regime before contributing to learning. We demonstrate the effectiveness of ProRes through pretraining experiments across various model scales, as well as normalization and initialization schemes. Comprehensive analysis shows that ProRes not only stabilizes pretraining but also introduces a unique optimization trajectory, leading to faster convergence, stronger generalization and better downstream performance. Our code is available at https://github.com/dandingsky/ProRes.
69aa4a84c4f6edddcc6c867e
Tianhao Chen
admin_assigned
2026-03-06T15:46:59.097Z
6475a2da04c82116f9ba5744
/avatars/d596935f0e67eb959b60a4787d6738c4.svg
Sky
dandingsky
user
dandingsky
69aa4a84c4f6edddcc6c867f
Xin Xu
claimed_verified
2026-03-09T10:37:59.654Z
64e2d169d2af12910d682130
https://cdn-avatars.huggingface.co/v1/production/uploads/64e2d169d2af12910d682130/VG8UdqJCJGc0K4G0P0XQP.jpeg
xuxin
xx18
user
xx18
69aa4a84c4f6edddcc6c8680
Lu Yin
69aa4a84c4f6edddcc6c8681
Hao Chen
69aa4a84c4f6edddcc6c8682
Yang Wang
69aa4a84c4f6edddcc6c8683
Shizhe Diao
69aa4a84c4f6edddcc6c8684
Can Yang
Progressive Residual Warmup for Language Model Pretraining
Transformer architectures serve as the backbone for most modern Large Language Models, therefore their pretraining stability and convergence speed are of central concern. Motivated by the logical dependency of sequentially stacked layers, we propose Progressive Residual Warmup (ProRes) for language model pretraining. ProRes implements an "early layer learns first" philosophy by multiplying each layer's residual with a scalar that gradually warms up from 0 to 1, with deeper layers taking longer warmup steps. In this way, deeper layers wait for early layers to settle into a more stable regime before contributing to learning. We demonstrate the effectiveness of ProRes through pretraining experiments across various model scales, as well as normalization and initialization schemes. Comprehensive analysis shows that ProRes not only stabilizes pretraining but also introduces a unique optimization trajectory, leading to faster convergence, stronger generalization and better downstream performance. Our code is available at https://github.com/dandingsky/ProRes.
Progressive Residual Warmup (ProRes) improves transformer language model pretraining by implementing gradual layer activation that stabilizes training and accelerates convergence.
Transformer architectures Large Language Models pretraining stability convergence speed progressive residual warmup residual connection layer activation optimization trajectory generalization downstream performance