FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions, aiming to enhance expert specialization beyond the single-dimension limit. We further introduce a bi-level sparse forward computation paradigm and a specialized routing mechanism to govern the activation. In addition, to obviate the prohibitive cost of training FineRMoE from scratch, we devise a generalized upcycling method to build FineRMoE in a cost-effective manner. Extensive experiments demonstrate the superior performance achieved by FineRMoE across ten standard benchmarks. Compared with the strongest baseline, FineRMoE achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.
69b8b6906b3f36c249fbc378
Ning Liao
admin_assigned
2026-03-17T08:44:23.621Z
6281ebe63e7734836f2f7724
https://cdn-avatars.huggingface.co/v1/production/uploads/6281ebe63e7734836f2f7724/lf5by3U79jW_mEKmJgVh2.jpeg
NingLiao
NingLiao
user
NingLiao
69b8b6906b3f36c249fbc379
Xiaoxing Wang
69b8b6906b3f36c249fbc37a
Xiaohan Qin
admin_assigned
2026-03-17T08:44:40.600Z
64452110e1fd8d65b2790bfc
https://cdn-avatars.huggingface.co/v1/production/uploads/64452110e1fd8d65b2790bfc/3NIsQFV9X44X88aLXR6eE.jpeg
Xiaohan Qin (SJTU) & (SII)
galaxy-1
user
galaxy-1
69b8b6906b3f36c249fbc37b
Junchi Yan
admin_assigned
2026-03-17T08:44:47.650Z
667289f903c802764985d8c6
/avatars/916befcbf0e52ce56be49617f31c7bb2.svg
Junchi Yan
Rethinker
user
Rethinker
FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
As revealed by the scaling law of fine-grained MoE, model performance ceases to be improved once the granularity of the intermediate dimension exceeds the optimal threshold, limiting further gains from single-dimension fine-grained design. To address this bottleneck, we propose FineRMoE (FineR-Grained MoE), an architecture that extends fine-grained expert design to both intermediate and output dimensions, aiming to enhance expert specialization beyond the single-dimension limit. We further introduce a bi-level sparse forward computation paradigm and a specialized routing mechanism to govern the activation. In addition, to obviate the prohibitive cost of training FineRMoE from scratch, we devise a generalized upcycling method to build FineRMoE in a cost-effective manner. Extensive experiments demonstrate the superior performance achieved by FineRMoE across ten standard benchmarks. Compared with the strongest baseline, FineRMoE achieves 6 times higher parameter efficiency, 281 times lower prefill latency, and 136 timese higher decoding throughput during inference.