On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
Multimodal Continual Instruction Tuning aims to continually enhance Large Vision Language Models (LVLMs) by learning from new data without forgetting previously acquired knowledge. Mixture of Experts (MoE) architectures naturally facilitate this by incrementally adding new experts and expanding routers while keeping the existing ones frozen. However, despite expert isolation, MoE-based continual learners still suffer from forgetting due to routing-drift: old-task tokens become mistakenly attracted to newly added experts, degrading performance on prior tasks. We analyze the failure mode at the token level and reveal the token's dilemma: ambiguous and old tokens in new-task data offer minimal learning benefit yet induce forgetting when routed to new experts, due to their ambiguous routing assignment during training. Motivated by this, we propose LLaVA-DyMoE, a dynamic MoE framework that incrementally expands the MoE with drift-aware token assignment. We characterize token types via their routing score distributions and apply targeted regularization. Specifically, a token-level assignment guidance steers ambiguous and old tokens away from new experts to preserve established routing patterns and alleviate routing-drift, while complementary routing score regularizations enforce expert-group separation and promote new-expert specialization. Extensive experiments demonstrate that our LLaVA-DyMoE effectively mitigates routing-drift-induced forgetting, achieving over a 7% gain in mean final accuracy and a 12% reduction in forgetting compared to baselines. The project page is https://zhaoc5.github.io/DyMoE.
69cb293122f9e9dce3fe1e8d
Chongyang Zhao
claimed_verified
2026-03-31T09:29:07.922Z
64c7671e77a6473ac325b6d8
/avatars/65434ae55207e785bd1ff2798d612fc2.svg
Chongyang Zhao
zhaoc5
user
zhaoc5
69cb293122f9e9dce3fe1e8e
Mingsong Li
admin_assigned
2026-03-31T09:39:16.483Z
67ade8d6d53a41cd62ee300c
https://cdn-avatars.huggingface.co/v1/production/uploads/67ade8d6d53a41cd62ee300c/ThE7nKEkVLmDxq_flbWdj.png
Mingsong_Li
Mingsong07
user
Mingsong07
69cb293122f9e9dce3fe1e8f
Haodong Lu
69cb293122f9e9dce3fe1e90
Dong Gong
claimed_verified
2026-03-31T09:28:58.961Z
6978aa1e2904a5468ef339f6
https://cdn-avatars.huggingface.co/v1/production/uploads/6978aa1e2904a5468ef339f6/cZ7Ae94lOrge_WBSzRSMG.jpeg
Dong Gong
dginf
user
dginf
On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models
Multimodal Continual Instruction Tuning aims to continually enhance Large Vision Language Models (LVLMs) by learning from new data without forgetting previously acquired knowledge. Mixture of Experts (MoE) architectures naturally facilitate this by incrementally adding new experts and expanding routers while keeping the existing ones frozen. However, despite expert isolation, MoE-based continual learners still suffer from forgetting due to routing-drift: old-task tokens become mistakenly attracted to newly added experts, degrading performance on prior tasks. We analyze the failure mode at the token level and reveal the token's dilemma: ambiguous and old tokens in new-task data offer minimal learning benefit yet induce forgetting when routed to new experts, due to their ambiguous routing assignment during training. Motivated by this, we propose LLaVA-DyMoE, a dynamic MoE framework that incrementally expands the MoE with drift-aware token assignment. We characterize token types via their routing score distributions and apply targeted regularization. Specifically, a token-level assignment guidance steers ambiguous and old tokens away from new experts to preserve established routing patterns and alleviate routing-drift, while complementary routing score regularizations enforce expert-group separation and promote new-expert specialization. Extensive experiments demonstrate that our LLaVA-DyMoE effectively mitigates routing-drift-induced forgetting, achieving over a 7% gain in mean final accuracy and a 12% reduction in forgetting compared to baselines. The project page is https://zhaoc5.github.io/DyMoE.
LLaVA-DyMoE addresses routing-drift-induced forgetting in multimodal continual instruction tuning by dynamically expanding mixture of experts with token-level assignment guidance and routing score regularizations.
Multimodal Continual Instruction Tuning Large Vision Language Models Mixture of Experts routing-drift token-level assignment guidance routing score regularizations expert-group separation new-expert specialization