ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning
Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks. Mixture-of-LoRAs models expand neural networks efficiently by routing each layer input to a small subset of specialized LoRAs of the layer. Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router. Despite their empirical promise, we observe that the routing weights are typically extremely imbalanced across LoRAs in practice, where only one or two LoRAs often dominate the routing weights. This essentially limits the number of effective LoRAs and thus severely hinders the expressive power of existing Mixture-of-LoRAs models. In this work, we attribute this weakness to the nature of learnable routing weights and rethink the fundamental design of the router. To address this critical issue, we propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix). Our key idea is using non-learnable routing weights to ensure all active LoRAs to be equally effective, with no LoRA dominating the routing weights. However, our routers cannot be trained directly via gradient descent due to our non-learnable routing weights. Hence, we further propose an unbiased gradient estimator for the router by employing the reinforce leave-one-out (RLOO) technique, where we regard the supervision loss as the reward and the router as the policy in reinforcement learning. Our gradient estimator also enables to scale up training compute to boost the predictive performance of our ReMix. Extensive experiments demonstrate that our proposed ReMix significantly outperform state-of-the-art parameter-efficient finetuning methods under a comparable number of activated parameters.
69b2398e206bad330cd7ffe0
Ruizhong Qiu
admin_assigned
2026-03-12T09:27:58.992Z
65370d95019de94263ad34a7
https://cdn-avatars.huggingface.co/v1/production/uploads/65370d95019de94263ad34a7/fH9SoJfX7yifXup0DvXkm.jpeg
Ruizhong Qiu
q-rz
user
q-rz
69b2398e206bad330cd7ffe1
Hanqing Zeng
admin_assigned
2026-03-12T09:28:06.582Z
6729196729fdd2b3fc360e21
/avatars/407f0660c8b2b63bf2b60aa9553bb2c6.svg
Hanqing Zeng
zimplex
user
zimplex
69b2398e206bad330cd7ffe2
Yinglong Xia
admin_assigned
2026-03-12T09:28:12.689Z
69121ebb0610df3fe9b801c2
/avatars/69fa06bba655874a91f57e223238a75f.svg
Yinglong Xia
yinglongxia
user
yinglongxia
69b2398e206bad330cd7ffe3
Yiwen Meng
69b2398e206bad330cd7ffe4
Ren Chen
69b2398e206bad330cd7ffe5
Jiarui Feng
69b2398e206bad330cd7ffe6
Dongqi Fu
admin_assigned
2026-03-12T09:29:25.080Z
68f32a44d3343fdfab09437c
/avatars/ebb78690bcab8b9ee1e07b3b5b401ae7.svg
DongqiFu
dongqifu
user
dongqifu
69b2398e206bad330cd7ffe7
Qifan Wang
69b2398e206bad330cd7ffe8
Jiayi Liu
69b2398e206bad330cd7ffe9
Jun Xiao
69b2398e206bad330cd7ffea
Xiangjun Fan
69b2398e206bad330cd7ffeb
Benyu Zhang
69b2398e206bad330cd7ffec
Hong Li
69b2398e206bad330cd7ffed
Zhining Liu
69b2398e206bad330cd7ffee
Hyunsik Yoo
69b2398e206bad330cd7ffef
Zhichen Zeng
admin_assigned
2026-03-12T09:29:53.793Z
66038f8e90c62bb38f06eb9f
/avatars/e67137b47f45f580bae7ea6472671359.svg
Zhichen Zeng
CharyZeng
user
CharyZeng
69b2398e206bad330cd7fff0
Tianxin Wei
admin_assigned
2026-03-12T09:29:41.272Z
647bc5022d27d3541df04a91
/avatars/e062092036da6c38f700362c0c5437c0.svg
Tianxin Wei
tianxinwei
user
tianxinwei
69b2398e206bad330cd7fff1
Hanghang Tong
ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning
Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks. Mixture-of-LoRAs models expand neural networks efficiently by routing each layer input to a small subset of specialized LoRAs of the layer. Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router. Despite their empirical promise, we observe that the routing weights are typically extremely imbalanced across LoRAs in practice, where only one or two LoRAs often dominate the routing weights. This essentially limits the number of effective LoRAs and thus severely hinders the expressive power of existing Mixture-of-LoRAs models. In this work, we attribute this weakness to the nature of learnable routing weights and rethink the fundamental design of the router. To address this critical issue, we propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix). Our key idea is using non-learnable routing weights to ensure all active LoRAs to be equally effective, with no LoRA dominating the routing weights. However, our routers cannot be trained directly via gradient descent due to our non-learnable routing weights. Hence, we further propose an unbiased gradient estimator for the router by employing the reinforce leave-one-out (RLOO) technique, where we regard the supervision loss as the reward and the router as the policy in reinforcement learning. Our gradient estimator also enables to scale up training compute to boost the predictive performance of our ReMix. Extensive experiments demonstrate that our proposed ReMix significantly outperform state-of-the-art parameter-efficient finetuning methods under a comparable number of activated parameters.
Researchers address imbalance in routing weights of Mixture-of-LoRAs models by proposing Reinforcement Routing (ReMix), which uses non-learnable weights and reinforcement learning techniques to improve model expressiveness and performance.
low-rank adapters parameter-efficient fine-tuning mixture-of-LoRAs routing weights reinforcement learning gradient estimator reinforce leave-one-out policy gradient neural network routing