Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
Low-Rank Adaptation (LoRA) is the prevailing approach for efficient large language model (LLM) fine-tuning. Building on this paradigm, recent studies have proposed alternative initialization strategies and architectural modifications, reporting substantial improvements over vanilla LoRA. However, these gains are often demonstrated under fixed or narrowly tuned hyperparameter settings, despite the known sensitivity of neural networks to training configurations. In this work, we systematically re-evaluate four representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches. Across mathematical and code generation tasks on diverse model scales, we find that different LoRA methods favor distinct learning rate ranges. Crucially, once learning rates are properly tuned, all methods achieve similar peak performance (within 1-2%), with only subtle rank-dependent behaviors. These results suggest that vanilla LoRA remains a competitive baseline and that improvements reported under single training configuration may not reflect consistent methodological advantages. Finally, a second-order analysis attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.
698639f72d626112378ad517
Yu-Ang Lee
claimed_verified
2026-02-11T11:20:09.127Z
6615752da15c52fa7ab3e2f7
/avatars/37e72cfb829a42630d229080ad8d60f3.svg
Lee
Speeeed
user
Speeeed
698639f72d626112378ad518
Ching-Yun Ko
698639f72d626112378ad519
Pin-Yu Chen
698639f72d626112378ad51a
Mi-Yen Yeh
Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning
Low-Rank Adaptation (LoRA) is the prevailing approach for efficient large language model (LLM) fine-tuning. Building on this paradigm, recent studies have proposed alternative initialization strategies and architectural modifications, reporting substantial improvements over vanilla LoRA. However, these gains are often demonstrated under fixed or narrowly tuned hyperparameter settings, despite the known sensitivity of neural networks to training configurations. In this work, we systematically re-evaluate four representative LoRA variants alongside vanilla LoRA through extensive hyperparameter searches. Across mathematical and code generation tasks on diverse model scales, we find that different LoRA methods favor distinct learning rate ranges. Crucially, once learning rates are properly tuned, all methods achieve similar peak performance (within 1-2%), with only subtle rank-dependent behaviors. These results suggest that vanilla LoRA remains a competitive baseline and that improvements reported under single training configuration may not reflect consistent methodological advantages. Finally, a second-order analysis attributes the differing optimal learning rate ranges to variations in the largest Hessian eigenvalue, aligning with classical learning theories.
Systematic evaluation of LoRA variants reveals that proper hyperparameter tuning eliminates performance differences between methods, with vanilla LoRA remaining competitive.
Low-Rank Adaptation large language models fine-tuning hyperparameter search learning rate Hessian eigenvalue