DR-LoRA: Dynamic Rank LoRA for Mixture-of-Experts Adaptation
Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning (PEFT), such as LoRA, is widely adopted to adapt pretrained MoE LLMs to downstream tasks. However, existing approaches assign identical LoRA ranks to all experts, overlooking the intrinsic functional specialization within MoE LLMs. This uniform allocation leads to resource mismatch, task-relevant experts are under-provisioned while less relevant ones receive redundant parameters. We propose a Dynamic Rank LoRA framework named DR-LoRA, which dynamically grows expert LoRA ranks during fine-tuning based on task-specific demands. DR-LoRA employs an Expert Saliency Scoring mechanism that integrates expert routing frequency and LoRA rank importance to quantify each expert's demand for additional capacity. Experts with higher saliency scores are prioritized for rank expansion, enabling the automatic formation of a heterogeneous rank distribution tailored to the target task. Experiments on multiple benchmarks demonstrate that DR-LoRA consistently outperforms standard LoRA and static allocation strategies under the same parameter budget, achieving superior task performance with more efficient parameter utilization.
6964607c138cc47cbd765259
Guanzhi Deng
6964607c138cc47cbd76525a
Bo Li
claimed_verified
2026-01-12T10:34:43.299Z
6582c482f3006507ea10302a
https://cdn-avatars.huggingface.co/v1/production/uploads/6582c482f3006507ea10302a/KbgSsq0FnbMngBcWPhIXi.jpeg
Bo Li
liboaccn
user
liboaccn
6964607c138cc47cbd76525b
Ronghao Chen
6964607c138cc47cbd76525c
Huacan Wang
claimed_verified
2026-01-15T15:07:27.943Z
6603d56ab4344a2b07cd6d21
/avatars/1569bb60166532317c85e80da722ba1c.svg
Huacan Wang
Huacan-Wang
user
Huacan-Wang
6964607c138cc47cbd76525d
Linqi Song
6964607c138cc47cbd76525e
Lijie Wen
DR-LoRA: Dynamic Rank LoRA for Mixture-of-Experts Adaptation
Mixture-of-Experts (MoE) has become a prominent paradigm for scaling Large Language Models (LLMs). Parameter-efficient fine-tuning (PEFT), such as LoRA, is widely adopted to adapt pretrained MoE LLMs to downstream tasks. However, existing approaches assign identical LoRA ranks to all experts, overlooking the intrinsic functional specialization within MoE LLMs. This uniform allocation leads to resource mismatch, task-relevant experts are under-provisioned while less relevant ones receive redundant parameters. We propose a Dynamic Rank LoRA framework named DR-LoRA, which dynamically grows expert LoRA ranks during fine-tuning based on task-specific demands. DR-LoRA employs an Expert Saliency Scoring mechanism that integrates expert routing frequency and LoRA rank importance to quantify each expert's demand for additional capacity. Experts with higher saliency scores are prioritized for rank expansion, enabling the automatic formation of a heterogeneous rank distribution tailored to the target task. Experiments on multiple benchmarks demonstrate that DR-LoRA consistently outperforms standard LoRA and static allocation strategies under the same parameter budget, achieving superior task performance with more efficient parameter utilization.
DR-LoRA dynamically adjusts LoRA ranks for experts in Mixture-of-Experts models based on task-specific demands, improving parameter efficiency and performance.
Mixture-of-Experts Large Language Models parameter-efficient fine-tuning LoRA dynamic rank allocation expert routing frequency expert saliency scoring heterogeneous rank distribution