ViT-AdaLA: Adapting Vision Transformers with Linear Attention
Vision Transformers (ViTs) based vision foundation models (VFMs) have achieved remarkable performance across diverse vision tasks, but suffer from quadratic complexity that limits scalability to long sequences. Existing linear attention approaches for ViTs are typically trained from scratch, requiring substantial computational resources, while linearization-based methods developed for large language model decoders do not transfer well to ViTs. To address these challenges, we propose ViT-AdaLA, a novel framework for effectively adapting and transferring prior knowledge from VFMs to linear attention ViTs. ViT-AdaLA consists of three stages: attention alignment, feature alignment, and supervised fine-tuning. In the attention alignment stage, we align vanilla linear attention with the original softmax-based attention in each block to approximate the behavior of softmax attention. However, residual approximation errors inevitably accumulate across layers. We mitigate this by fine-tuning the linearized ViT to align its final-layer features with a frozen softmax VFM teacher. Finally, the adapted prior knowledge is transferred to downstream tasks through supervised fine-tuning. Extensive experiments on classification and segmentation tasks demonstrate the effectiveness and generality of ViT-AdaLA over various state-of-the-art linear attention counterpart.
69ba41d55d2b9c28ea09f233
Yifan Li
69ba41d55d2b9c28ea09f234
Seunghyun Yoon
69ba41d55d2b9c28ea09f235
Viet Dac Lai
69ba41d55d2b9c28ea09f236
Franck Dernoncourt
claimed_verified
2026-03-18T08:35:51.702Z
62c5947524171688a9feb992
/avatars/5a151713b9eae8dc566f5957acee3475.svg
Franck Dernoncourt
Franck-Dernoncourt
user
Franck-Dernoncourt
69ba41d55d2b9c28ea09f237
Jason Kuen
admin_assigned
2026-03-18T10:19:20.422Z
65384eaeef7d2bb8c4baf1b2
/avatars/b9217561a59309d6b0ddb417324c6cad.svg
Jason Kuen
xternalz
user
xternalz
69ba41d55d2b9c28ea09f238
Yu Kong
69ba41d55d2b9c28ea09f239
Trung Bui
ViT-AdaLA: Adapting Vision Transformers with Linear Attention
Vision Transformers (ViTs) based vision foundation models (VFMs) have achieved remarkable performance across diverse vision tasks, but suffer from quadratic complexity that limits scalability to long sequences. Existing linear attention approaches for ViTs are typically trained from scratch, requiring substantial computational resources, while linearization-based methods developed for large language model decoders do not transfer well to ViTs. To address these challenges, we propose ViT-AdaLA, a novel framework for effectively adapting and transferring prior knowledge from VFMs to linear attention ViTs. ViT-AdaLA consists of three stages: attention alignment, feature alignment, and supervised fine-tuning. In the attention alignment stage, we align vanilla linear attention with the original softmax-based attention in each block to approximate the behavior of softmax attention. However, residual approximation errors inevitably accumulate across layers. We mitigate this by fine-tuning the linearized ViT to align its final-layer features with a frozen softmax VFM teacher. Finally, the adapted prior knowledge is transferred to downstream tasks through supervised fine-tuning. Extensive experiments on classification and segmentation tasks demonstrate the effectiveness and generality of ViT-AdaLA over various state-of-the-art linear attention counterpart.
ViT-AdaLA adapts vision foundation models to linear attention Vision Transformers through attention alignment, feature alignment, and supervised fine-tuning to overcome quadratic complexity limitations.
Vision Transformers vision foundation models linear attention softmax attention attention alignment feature alignment supervised fine-tuning residual approximation errors downstream tasks