Improving Recursive Transformers with Mixture of LoRAs
Parameter sharing in recursive transformers reduces model size but collapses layer-wise expressivity. We propose Mixture of LoRAs (MoL), a lightweight conditional-computation mechanism that inserts Low-Rank Adaptation (LoRA) experts inside a shared feed-forward network (FFN). MoL enables token-conditional weight-space modulation of the shared FFN without untying backbone parameters, unlike prior approaches that add fixed or externally attached adapters. We pretrain a modernised recursive architecture, ModernALBERT, integrating rotary embeddings, GeGLU, FlashAttention, and a distillation-based initialisation. Across GLUE, SQuAD-v2, and BEIR, ModernALBERT (50M--120M) achieves state-of-the-art performance among compact models and surpasses larger fully parameterised baselines. We also propose an expert-merging procedure that compresses MoL into a single adapter at inference while preserving accuracy, enabling efficient deployment. Our results show that conditional weight-space modulation effectively restores the expressivity lost under aggressive parameter sharing in recursive transformers.
6945ea06316ca45c45728316
Mohammadmahdi Nouriborji
claimed_verified
2025-12-29T14:25:09.002Z
62b615f3d25e06ad3e8e7db8
https://cdn-avatars.huggingface.co/v1/production/uploads/62b615f3d25e06ad3e8e7db8/ClGP0pzOKsivZoPeaPZ8P.png
Mohammadmahdi Nouriborji
mohammadmahdinouri
user
mohammadmahdinouri
6945ea06316ca45c45728317
Morteza Rohanian
6945ea06316ca45c45728318
Omid Rohanian
claimed_verified
2025-12-22T10:59:29.261Z
62fb7f27f60cc3ae073a0c1b
https://cdn-avatars.huggingface.co/v1/production/uploads/1662630571545-62fb7f27f60cc3ae073a0c1b.jpeg
Omid Rohanian
omidrohanian
user
omidrohanian
Improving Recursive Transformers with Mixture of LoRAs
Parameter sharing in recursive transformers reduces model size but collapses layer-wise expressivity. We propose Mixture of LoRAs (MoL), a lightweight conditional-computation mechanism that inserts Low-Rank Adaptation (LoRA) experts inside a shared feed-forward network (FFN). MoL enables token-conditional weight-space modulation of the shared FFN without untying backbone parameters, unlike prior approaches that add fixed or externally attached adapters. We pretrain a modernised recursive architecture, ModernALBERT, integrating rotary embeddings, GeGLU, FlashAttention, and a distillation-based initialisation. Across GLUE, SQuAD-v2, and BEIR, ModernALBERT (50M--120M) achieves state-of-the-art performance among compact models and surpasses larger fully parameterised baselines. We also propose an expert-merging procedure that compresses MoL into a single adapter at inference while preserving accuracy, enabling efficient deployment. Our results show that conditional weight-space modulation effectively restores the expressivity lost under aggressive parameter sharing in recursive transformers.
Mixture of LoRAs within a shared feed-forward network restores expressivity in parameter-shared recursive transformers, achieving state-of-the-art performance with compact models.
parameter sharing recursive transformers Mixture of LoRAs LoRA feed-forward network token-conditional weight-space modulation backbone parameters ModernALBERT rotary embeddings GeGLU FlashAttention distillation-based initialisation GLUE SQuAD-v2 BEIR expert-merging procedure