Accurate and Efficient Low-Rank Model Merging in Core Space
In this paper, we address the challenges associated with merging low-rank
adaptations of large neural networks. With the rise of parameter-efficient
adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning
has become more accessible. While fine-tuning models with LoRA is highly
efficient, existing merging methods often sacrifice this efficiency by merging
fully-sized weight matrices. We propose the Core Space merging framework, which
enables the merging of LoRA-adapted models within a common alignment basis,
thereby preserving the efficiency of low-rank adaptation while substantially
improving accuracy across tasks. We further provide a formal proof that
projection into Core Space ensures no loss of information and provide a
complexity analysis showing the efficiency gains. Extensive empirical results
demonstrate that Core Space significantly improves existing merging techniques
and achieves state-of-the-art results on both vision and language tasks while
utilizing a fraction of the computational resources. Codebase is available at
https://github.com/apanariello4/core-space-merging.
68d2db0d0e215259d193b11b
Aniello Panariello
claimed_verified
2025-09-24T13:53:45.230Z
651e940a081a74588de506c0
/avatars/ae34ee47818135695c0edf01b0b1508c.svg
Aniello Panariello
nellopan
user
nellopan
68d2db0d0e215259d193b11c
Daniel Marczak
claimed_verified
2025-09-24T13:54:11.809Z
65a5358ddb5c00652ef24c8d
/avatars/d50b6297584c9b4c2ccd93e64477b940.svg
Daniel Marczak
danielm1405
user
danielm1405
68d2db0d0e215259d193b11d
Simone Magistri
68d2db0d0e215259d193b11e
Angelo Porrello
68d2db0d0e215259d193b11f
Bartłomiej Twardowski
68d2db0d0e215259d193b120
Andrew D. Bagdanov
68d2db0d0e215259d193b121
Simone Calderara
68d2db0d0e215259d193b122
Joost van de Weijer
Accurate and Efficient Low-Rank Model Merging in Core Space
In this paper, we address the challenges associated with merging low-rank
adaptations of large neural networks. With the rise of parameter-efficient
adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning
has become more accessible. While fine-tuning models with LoRA is highly
efficient, existing merging methods often sacrifice this efficiency by merging
fully-sized weight matrices. We propose the Core Space merging framework, which
enables the merging of LoRA-adapted models within a common alignment basis,
thereby preserving the efficiency of low-rank adaptation while substantially
improving accuracy across tasks. We further provide a formal proof that
projection into Core Space ensures no loss of information and provide a
complexity analysis showing the efficiency gains. Extensive empirical results
demonstrate that Core Space significantly improves existing merging techniques
and achieves state-of-the-art results on both vision and language tasks while
utilizing a fraction of the computational resources. Codebase is available at
https://github.com/apanariello4/core-space-merging.
Core Space merging framework improves the accuracy and efficiency of merging low-rank adapted models across tasks without significant computational cost.
Low-Rank Adaptation LoRA Core Space parameter-efficient adaptation merging methods weight matrices formal proof complexity analysis state-of-the-art results vision tasks language tasks