Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers
Applying weight decay (WD) to matrix layers is standard practice in large-language-model pretraining. Prior work suggests that stochastic gradient noise induces a Brownian-like expansion of the weight matrices W, whose growth is counteracted by WD, leading to a WD-noise equilibrium with a certain weight norm ||W||. In this work, we view the equilibrium norm as a harmful artifact of the training procedure, and address it by introducing learnable multipliers to learn the optimal scale. First, we attach a learnable scalar multiplier to W and confirm that the WD-noise equilibrium norm is suboptimal: the learned scale adapts to data and improves performance. We then argue that individual row and column norms are similarly constrained, and free their scale by introducing learnable per-row and per-column multipliers. Our method can be viewed as a learnable, more expressive generalization of muP multipliers. It outperforms a well-tuned muP baseline, reduces the computational overhead of multiplier tuning, and surfaces practical questions such as forward-pass symmetries and the width-scaling of the learned multipliers. Finally, we validate learnable multipliers with both Adam and Muon optimizers, where it shows improvement in downstream evaluations matching the improvement of the switching from Adam to Muon.
69608e7c5b7998385e639583
Maksim Velikanov
admin_assigned
2026-01-09T15:49:44.975Z
64670db15993aa7666cc6022
/avatars/b68caad7e987c095b0cab4d9035aac25.svg
Maksim Velikanov
yellowvm
user
yellowvm
69608e7c5b7998385e639584
Ilyas Chahed
claimed_verified
2026-01-09T15:46:04.314Z
6697a9fb6d173ec7382e0392
https://cdn-avatars.huggingface.co/v1/production/uploads/6697a9fb6d173ec7382e0392/Q_myrIBbdWI3RmEvEHrtQ.jpeg
Ilyas Chahed
IChahed
user
IChahed
69608e7c5b7998385e639585
Jingwei Zuo
claimed_verified
2026-01-09T08:34:49.874Z
6460c3811db65f878513bcaf
https://cdn-avatars.huggingface.co/v1/production/uploads/6460c3811db65f878513bcaf/0C3SoPO0bqTKtnWIu1WtG.jpeg
Jingwei Zuo
JingweiZuo
user
JingweiZuo
69608e7c5b7998385e639586
Dhia Eddine Rhaiem
69608e7c5b7998385e639587
Younes Belkada
claimed_verified
2026-01-09T08:34:47.914Z
62441d1d9fdefb55a0b7d12c
https://cdn-avatars.huggingface.co/v1/production/uploads/1648631057413-noauth.png
Younes B
ybelkada
user
ybelkada
69608e7c5b7998385e639588
Hakim Hacid
admin_assigned
2026-01-09T15:49:56.651Z
6471d727a2b0a376b8b6a4ed
/avatars/aedda547f2ca40dfa898e76be787952f.svg
Hakim Hacid
HakimHacid
user
HakimHacid
Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers
Applying weight decay (WD) to matrix layers is standard practice in large-language-model pretraining. Prior work suggests that stochastic gradient noise induces a Brownian-like expansion of the weight matrices W, whose growth is counteracted by WD, leading to a WD-noise equilibrium with a certain weight norm ||W||. In this work, we view the equilibrium norm as a harmful artifact of the training procedure, and address it by introducing learnable multipliers to learn the optimal scale. First, we attach a learnable scalar multiplier to W and confirm that the WD-noise equilibrium norm is suboptimal: the learned scale adapts to data and improves performance. We then argue that individual row and column norms are similarly constrained, and free their scale by introducing learnable per-row and per-column multipliers. Our method can be viewed as a learnable, more expressive generalization of muP multipliers. It outperforms a well-tuned muP baseline, reduces the computational overhead of multiplier tuning, and surfaces practical questions such as forward-pass symmetries and the width-scaling of the learned multipliers. Finally, we validate learnable multipliers with both Adam and Muon optimizers, where it shows improvement in downstream evaluations matching the improvement of the switching from Adam to Muon.
Learnable multipliers are introduced to address weight decay-induced normalization artifacts in large language model training, outperforming traditional methods while reducing computational overhead.
weight decay stochastic gradient noise Brownian-like expansion WD-noise equilibrium learnable multipliers matrix layers weight norm muP multipliers Adam optimizer Muon optimizer