Scaling Embeddings Outperforms Scaling Experts in Language Models
While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.
697c3801a67238fac88cc1b1
Hong Liu
697c3801a67238fac88cc1b2
Jiaqi Zhang
697c3801a67238fac88cc1b3
Chao Wang
697c3801a67238fac88cc1b4
Xing Hu
697c3801a67238fac88cc1b5
Linkun Lyu
claimed_verified
2026-02-02T16:59:30.907Z
64cc5eb3942890af937473c4
/avatars/38b71030d6fc5592565a0d286a0c2ec4.svg
Linkun Lyu
llk2why
user
llk2why
697c3801a67238fac88cc1b6
Jiaqi Sun
697c3801a67238fac88cc1b7
Xurui Yang
697c3801a67238fac88cc1b8
Bo Wang
697c3801a67238fac88cc1b9
Fengcun Li
697c3801a67238fac88cc1ba
Yulei Qian
697c3801a67238fac88cc1bb
Lingtong Si
697c3801a67238fac88cc1bc
Yerui Sun
697c3801a67238fac88cc1bd
Rumei Li
697c3801a67238fac88cc1be
Peng Pei
697c3801a67238fac88cc1bf
Yuchen Xie
697c3801a67238fac88cc1c0
Xunliang Cai
Scaling Embeddings Outperforms Scaling Experts in Language Models
While Mixture-of-Experts (MoE) architectures have become the standard for sparsity scaling in large language models, they increasingly face diminishing returns and system-level bottlenecks. In this work, we explore embedding scaling as a potent, orthogonal dimension for scaling sparsity. Through a comprehensive analysis and experiments, we identify specific regimes where embedding scaling achieves a superior Pareto frontier compared to expert scaling. We systematically characterize the critical architectural factors governing this efficacy -- ranging from parameter budgeting to the interplay with model width and depth. Moreover, by integrating tailored system optimizations and speculative decoding, we effectively convert this sparsity into tangible inference speedups. Guided by these insights, we introduce LongCat-Flash-Lite, a 68.5B parameter model with ~3B activated trained from scratch. Despite allocating over 30B parameters to embeddings, LongCat-Flash-Lite not only surpasses parameter-equivalent MoE baselines but also exhibits exceptional competitiveness against existing models of comparable scale, particularly in agentic and coding domains.
Embedding scaling offers superior sparsity scaling compared to expert scaling in large language models, enabling efficient inference through system optimizations and speculative decoding.
Mixture-of-Experts sparsity scaling embedding scaling Pareto frontier parameter budgeting model width model depth system optimizations speculative decoding LongCat-Flash-Lite