SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose SPARKLING (balancing {S}ignal {P}reservation {A}nd symmet{R}y brea{K}ing for width-progressive {L}earn{ING}), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state resetting and learning rate re-warmup. Extensive experiments on Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under 2times width expansion.
69819059ce18b186280963b8
Qifan Yu
claimed_verified
2026-02-09T08:36:14.707Z
650b03e64e6a363198b8af6b
/avatars/cd427065d23039b3228257c91f1ce447.svg
Qifan Yu
CauchyLovesU
user
CauchyLovesU
69819059ce18b186280963b9
Xinyu Ma
claimed_verified
2026-02-03T10:02:56.011Z
6437a5a4961bb61e463856af
/avatars/fb9feb3e45001af42f0e19e236f784bb.svg
Xinyu Ma
EternalFlame
user
EternalFlame
69819059ce18b186280963ba
Zhijian Zhuo
69819059ce18b186280963bb
Minrui Wang
69819059ce18b186280963bc
Deyi Liu
69819059ce18b186280963bd
Shiyi Zhan
69819059ce18b186280963be
Yiyuan Ma
69819059ce18b186280963bf
Liang Xiang
69819059ce18b186280963c0
Xingyan Bin
69819059ce18b186280963c1
Di He
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose SPARKLING (balancing {S}ignal {P}reservation {A}nd symmet{R}y brea{K}ing for width-progressive {L}earn{ING}), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state resetting and learning rate re-warmup. Extensive experiments on Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under 2times width expansion.
SPARKLING is a framework for mid-stage width expansion in deep learning models that maintains signal preservation and breaks symmetry to stabilize training and reduce computational costs.
progressive learning width expansion training instabilities activation statistics loss spikes gradient symmetry signal preservation RMS-scale consistency asymmetric optimizer state resetting learning rate re-warmup Mixture-of-Experts computational savings