Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers
The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method.
6978295b026bdf0473116db5
Zecheng Tang
claimed_verified
2026-01-27T09:42:48.253Z
64096ef79e9f790c905b846d
https://cdn-avatars.huggingface.co/v1/production/uploads/64096ef79e9f790c905b846d/hVzw656lXdzbCxTtnheud.jpeg
Zecheng Tang
ZetangForward
user
ZetangForward
6978295b026bdf0473116db6
Quantong Qiu
admin_assigned
2026-01-27T14:27:08.590Z
6732fb1d24b316be87acaafe
https://cdn-avatars.huggingface.co/v1/production/uploads/6732fb1d24b316be87acaafe/BzD8HL4vhh3mfeSF3rm_1.jpeg
Quantong Qiu
QQTang1223
user
QQTang1223
6978295b026bdf0473116db7
Yi Yang
6978295b026bdf0473116db8
Zhiyi Hong
6978295b026bdf0473116db9
Haiya Xiang
6978295b026bdf0473116dba
Kebin Liu
claimed_verified
2026-01-28T11:32:06.877Z
67e7e66d0d9434ad14575739
/avatars/5cf522a690519c22b65b959c9e0ca8d2.svg
Kebin Liu
wanderhzz
user
wanderhzz
6978295b026bdf0473116dbb
Qingqing Dang
admin_assigned
2026-01-27T14:27:50.734Z
68cce9276e4618473d590342
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/UqBWbWZ58EPwJ5iRUsJsB.png
Qingqing Dang
DaisyGrace
user
DaisyGrace
6978295b026bdf0473116dbc
Juntao Li
admin_assigned
2026-01-27T14:27:56.908Z
6670e285b0c03c4e9d6e0985
https://cdn-avatars.huggingface.co/v1/production/uploads/6670e285b0c03c4e9d6e0985/j9Zr-lOtrRmpFz5f4x420.jpeg
Juntao Li
douvleplus
user
douvleplus
6978295b026bdf0473116dbd
Min Zhang
Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers
The quadratic complexity of standard attention mechanisms poses a significant scalability bottleneck for large language models (LLMs) in long-context scenarios. While hybrid attention strategies that combine sparse and full attention within a single model offer a viable solution, they typically employ static computation ratios (i.e., fixed proportions of sparse versus full attention) and fail to adapt to the varying sparsity sensitivities of downstream tasks during inference. To address this issue, we propose Elastic Attention, which allows the model to dynamically adjust its overall sparsity based on the input. This is achieved by integrating a lightweight Attention Router into the existing pretrained model, which dynamically assigns each attention head to different computation modes. Within only 12 hours of training on 8xA800 GPUs, our method enables models to achieve both strong performance and efficient inference. Experiments across three long-context benchmarks on widely-used LLMs demonstrate the superiority of our method.
Elastic Attention enables dynamic adjustment of attention sparsity during inference by integrating a lightweight Attention Router into pretrained models, achieving efficient long-context processing.
standard attention mechanisms sparse attention full attention hybrid attention strategies attention router attention heads long-context scenarios pretrained models computational efficiency