Janus: Disaggregating Attention and Experts for Scalable MoE Inference
Large Mixture-of-Experts (MoE) model inference is challenging due to high resource demands and dynamic workloads. Existing solutions often deploy the entire model as a single monolithic unit, which applies a unified resource configuration to both attention and expert modules despite their different requirements, leading to limited scalability and resource inefficiency. In this paper, we propose Janus, a scalable MoE inference system that disaggregates attention and experts on separate GPU sub-clusters, enabling each module to be managed and scaled independently. Janus incorporates three key designs for efficient, disaggregated MoE inference. First, it proposes an adaptive two-phase communication scheme that exploits intra- and inter-node bandwidth hierarchies for low-latency data exchange. Second, motivated by the memory-bound nature of MoE modules, Janus introduces a lightweight scheduler and implements it as a GPU kernel to balance the number of activated experts across GPUs at minimal overhead, thereby reducing inference latency. Third, Janus performs fine-grained resource management to dynamically adjust expert placement and independently scale attention and MoE resources to improve overall efficiency. Evaluation shows Janus achieves up to 3.9 higher perGPU throughput than state-of-the-art systems while meeting per-token latency requirements.
69426c435d5b2dc105274a29
Zhexiang Zhang
69426c435d5b2dc105274a2a
Ye Wang
claimed_verified
2025-12-17T10:07:09.621Z
69426dad01bb07a51208eead
/avatars/0828b4fae4ff120604f65677782aa311.svg
Ye Wang
yeyeyewang
user
yeyeyewang
69426c435d5b2dc105274a2b
Xiangyu Wang
claimed_verified
2025-12-17T10:07:17.583Z
668b818485624c0310c8ef7e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/ccdmmU05ydpZVaPU6BthZ.jpeg
wang
v3nividiv1ci
user
v3nividiv1ci
69426c435d5b2dc105274a2c
Yumiao Zhao
claimed_verified
2025-12-17T10:07:07.754Z
68e20d7f950469da930e9691
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/B2rB1n4DIazg7QT4GQiAu.jpeg
Yumiao Zhao
FunnyBoyMax
user
FunnyBoyMax
69426c435d5b2dc105274a2d
Jingzhe Jiang
69426c435d5b2dc105274a2e
Qizhen Weng
69426c435d5b2dc105274a2f
Shaohuai Shi
69426c435d5b2dc105274a30
Yin Chen
69426c435d5b2dc105274a31
Minchen Yu
Janus: Disaggregating Attention and Experts for Scalable MoE Inference
Large Mixture-of-Experts (MoE) model inference is challenging due to high resource demands and dynamic workloads. Existing solutions often deploy the entire model as a single monolithic unit, which applies a unified resource configuration to both attention and expert modules despite their different requirements, leading to limited scalability and resource inefficiency. In this paper, we propose Janus, a scalable MoE inference system that disaggregates attention and experts on separate GPU sub-clusters, enabling each module to be managed and scaled independently. Janus incorporates three key designs for efficient, disaggregated MoE inference. First, it proposes an adaptive two-phase communication scheme that exploits intra- and inter-node bandwidth hierarchies for low-latency data exchange. Second, motivated by the memory-bound nature of MoE modules, Janus introduces a lightweight scheduler and implements it as a GPU kernel to balance the number of activated experts across GPUs at minimal overhead, thereby reducing inference latency. Third, Janus performs fine-grained resource management to dynamically adjust expert placement and independently scale attention and MoE resources to improve overall efficiency. Evaluation shows Janus achieves up to 3.9 higher perGPU throughput than state-of-the-art systems while meeting per-token latency requirements.
Janus is a scalable Mixture-of-Experts (MoE) inference system that disaggregates attention and expert modules for independent scaling, improving throughput and latency.
Mixture-of-Experts MoE GPU sub-clusters adaptive two-phase communication lightweight scheduler GPU kernel fine-grained resource management expert placement per-GPU throughput per-token latency