MARS-M: When Variance Reduction Meets Matrices
Matrix-based preconditioned optimizers, such as Muon, have recently been
shown to be more efficient than scalar-based optimizers for training
large-scale neural networks, including large language models (LLMs). On the
other hand, recent benchmarks on optimizers for LLM pre-training have
demonstrated that variance-reduction techniques such as MARS can achieve
substantial speedups over standard optimizers that do not employ variance
reduction. In this paper, to achieve the best of both worlds, we introduce
MARS-M, a new optimizer that integrates the variance reduction technique in
MARS with Muon. Under standard regularity conditions, we prove that Muon-M
converges to a first-order stationary point at a rate of
mathcal{O}(T^{-1/3}), which improves upon
mathcal{O}(T^{-1/4}) rate attained by Muon. Our empirical results on
language modeling and computer vision tasks demonstrate that MARS-M
consistently yields lower losses and improved performance across various
downstream benchmarks. The implementation of MARS-M is available at
https://github.com/AGI-Arena/MARS/MARS_M.
69002c5e22d452aac6dd42bb
Yifeng Liu
claimed_verified
2025-10-28T15:36:44.528Z
653d276681f52ceb4d12bd85
/avatars/56601a25e5f883a8f6dc15f6fd9dcc57.svg
Yifeng Liu
Lewis-Lau
user
Lewis-Lau
69002c5e22d452aac6dd42bc
Angela Yuan
69002c5e22d452aac6dd42bd
Quanquan Gu
MARS-M: When Variance Reduction Meets Matrices
Matrix-based preconditioned optimizers, such as Muon, have recently been
shown to be more efficient than scalar-based optimizers for training
large-scale neural networks, including large language models (LLMs). On the
other hand, recent benchmarks on optimizers for LLM pre-training have
demonstrated that variance-reduction techniques such as MARS can achieve
substantial speedups over standard optimizers that do not employ variance
reduction. In this paper, to achieve the best of both worlds, we introduce
MARS-M, a new optimizer that integrates the variance reduction technique in
MARS with Muon. Under standard regularity conditions, we prove that Muon-M
converges to a first-order stationary point at a rate of
mathcal{O}(T^{-1/3}), which improves upon
mathcal{O}(T^{-1/4}) rate attained by Muon. Our empirical results on
language modeling and computer vision tasks demonstrate that MARS-M
consistently yields lower losses and improved performance across various
downstream benchmarks. The implementation of MARS-M is available at
https://github.com/AGI-Arena/MARS/MARS_M.
MARS-M, a new optimizer combining Muon and MARS techniques, achieves faster convergence and better performance in large-scale neural network training.
Muon MARS variance reduction first-order stationary point language modeling computer vision