REAM: Merging Improves Pruning of Experts in LLMs
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
69d5455fb9b236b710ae7f71
Saurav Jha
69d5455fb9b236b710ae7f72
Maryam Hashemzadeh
69d5455fb9b236b710ae7f73
Ali Saheb Pasand
69d5455fb9b236b710ae7f74
Ali Parviz
69d5455fb9b236b710ae7f75
Min-Joong Lee
69d5455fb9b236b710ae7f76
Boris Knyazev
claimed_verified
2026-04-08T16:34:23.143Z
63e16165f0039731dfdd442a
/avatars/37cf99dc016c291c800f60d260173482.svg
Boris Knyazev
bknyaz
user
bknyaz
REAM: Merging Improves Pruning of Experts in LLMs
Mixture-of-Experts (MoE) large language models (LLMs) are among the top-performing architectures. The largest models, often with hundreds of billions of parameters, pose significant memory challenges for deployment. Traditional approaches to reduce memory requirements include weight pruning and quantization. Motivated by the Router-weighted Expert Activation Pruning (REAP) that prunes experts, we propose a novel method, Router-weighted Expert Activation Merging (REAM). Instead of removing experts, REAM groups them and merges their weights, better preserving original performance. We evaluate REAM against REAP and other baselines across multiple MoE LLMs on diverse multiple-choice (MC) question answering and generative (GEN) benchmarks. Our results reveal a trade-off between MC and GEN performance that depends on the mix of calibration data. By controlling the mix of general, math and coding data, we examine the Pareto frontier of this trade-off and show that REAM often outperforms the baselines and in many cases is comparable to the original uncompressed models.
Router-weighted Expert Activation Merging (REAM) is proposed as a novel method for reducing memory requirements in Mixture-of-Experts large language models by grouping and merging expert weights instead of pruning them, achieving performance comparable to uncompressed models while maintaining efficiency.
Mixture-of-Experts large language models weight pruning quantization Router-weighted Expert Activation Pruning Router-weighted Expert Activation Merging expert activation merging calibration data Pareto frontier