MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head
While the Transformer architecture dominates many fields, its quadratic self-attention complexity hinders its use in large-scale applications. Linear attention offers an efficient alternative, but its direct application often degrades performance, with existing fixes typically re-introducing computational overhead through extra modules (e.g., depthwise separable convolution) that defeat the original purpose. In this work, we identify a key failure mode in these methods: global context collapse, where the model loses representational diversity. To address this, we propose Multi-Head Linear Attention (MHLA), which preserves this diversity by computing attention within divided heads along the token dimension. We prove that MHLA maintains linear complexity while recovering much of the expressive power of softmax attention, and verify its effectiveness across multiple domains, achieving a 3.6\% improvement on ImageNet classification, a 6.3\% gain on NLP, a 12.6\% improvement on image generation, and a 41\% enhancement on video generation under the same time complexity.
6965c791fc8c4ecc02c7f9d3
Kewei Zhang
claimed_verified
2026-01-13T08:23:45.745Z
659698e6f67e8fb2a5985445
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/6jxohd_DsLVfSnWgKMGrn.jpeg
Kewei Zhang
xiwenyoumu
user
xiwenyoumu
6965c791fc8c4ecc02c7f9d4
Ye Huang
6965c791fc8c4ecc02c7f9d5
Yufan Deng
claimed_verified
2026-01-13T08:23:39.576Z
68fce03ed1d0efce7ca87075
https://cdn-avatars.huggingface.co/v1/production/uploads/68fce03ed1d0efce7ca87075/GRKTeVIaLZD_M-KoJE8YF.png
yfdeng
yfdeng10
user
yfdeng10
6965c791fc8c4ecc02c7f9d6
Jincheng Yu
6965c791fc8c4ecc02c7f9d7
Junsong Chen
6965c791fc8c4ecc02c7f9d8
Huan Ling
6965c791fc8c4ecc02c7f9d9
Enze Xie
6965c791fc8c4ecc02c7f9da
Daquan Zhou
MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head
While the Transformer architecture dominates many fields, its quadratic self-attention complexity hinders its use in large-scale applications. Linear attention offers an efficient alternative, but its direct application often degrades performance, with existing fixes typically re-introducing computational overhead through extra modules (e.g., depthwise separable convolution) that defeat the original purpose. In this work, we identify a key failure mode in these methods: global context collapse, where the model loses representational diversity. To address this, we propose Multi-Head Linear Attention (MHLA), which preserves this diversity by computing attention within divided heads along the token dimension. We prove that MHLA maintains linear complexity while recovering much of the expressive power of softmax attention, and verify its effectiveness across multiple domains, achieving a 3.6\% improvement on ImageNet classification, a 6.3\% gain on NLP, a 12.6\% improvement on image generation, and a 41\% enhancement on video generation under the same time complexity.
Multi-Head Linear Attention addresses the performance degradation in linear attention by preserving representational diversity through head-wise token dimension computation, maintaining linear complexity while recovering softmax attention's expressive power across multiple domains.
Transformer architecture self-attention linear attention global context collapse Multi-Head Linear Attention token dimension softmax attention ImageNet classification NLP image generation video generation