The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain-invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. This inherent bias also indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance.
69606e2d5b7998385e6394a1
Yan Wang
claimed_verified
2026-01-09T08:35:15.313Z
65d76cc5b9b7b8bf88faa916
/avatars/d95232cd0c307efab6197ade1a66190b.svg
Yan Wang
YanAdjeNole
user
YanAdjeNole
69606e2d5b7998385e6394a2
Yitao Xu
69606e2d5b7998385e6394a3
Nanhan Shen
69606e2d5b7998385e6394a4
Jinyan Su
69606e2d5b7998385e6394a5
Jimin Huang
claimed_verified
2026-01-12T11:08:32.732Z
63b58ed5889aa6707f0bb0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/63b58ed5889aa6707f0bb0f4/znl74_aMswlV8VtHrfj3G.jpeg
Jimin Huang
jiminHuang
user
jiminHuang
69606e2d5b7998385e6394a6
Zining Zhu
The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain-invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. This inherent bias also indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance.
Research challenges the assumption of domain specialization in Mixture of Experts models by identifying a persistent central committee of experts that dominates routing behavior across different domains and architectures.
Mixture of Experts sparse routing COMMITTEEAUDIT domain specialization Standing Committee expert groups routing behavior MMLU benchmark load-balancing losses expert utilization