SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
The quadratic complexity of full attention limits efficient long-context processing in large language models (LLMs). Sparse attention mitigates this cost by restricting each query to attend to a subset of previous tokens; however, training-free approaches often lead to severe performance degradation. Native sparse-attention methods (e.g., NSA, MoBA) alleviate this issue, yet exhibit a critical paradox: they produce lower attention sparsity than full-attention models, despite aiming to approximate full attention, which may constrain their effectiveness. We attribute this paradox to gradient update deficiency: low-ranked key-value pairs excluded during sparse training receive neither forward contribution nor backward gradients, and thus never learn proper suppression. To overcome this limitation, we propose SSA (Sparse Sparse Attention), a unified training framework that considers both sparse and full attention and enforces bidirectional alignment at every layer. This design preserves gradient flow to all tokens while explicitly encouraging sparse-attention outputs to align with their full-attention counterparts, thereby promoting stronger sparsity. As a result, SSA achieves state-of-the-art performance under both sparse and full attention inference across multiple commonsense benchmarks. Furthermore, SSA enables models to adapt smoothly to varying sparsity budgets; performance improves consistently as more tokens are allowed to attend, supporting flexible compute-performance trade-offs at inference time. Finally, we show that native sparse-attention training surprisingly improves long-context extrapolation by mitigating the over-allocation of attention values in sink areas, with SSA demonstrating the strongest extrapolation capability.
69269351243b2216fb75cb22
Zhenyi Shen
claimed_verified
2025-11-26T11:31:23.403Z
641b31a4ec5b871c0bcd6932
/avatars/bad42279ad99918b0846053f2fa95ac8.svg
Zhenyi Shen
zen-E
user
zen-E
69269351243b2216fb75cb23
Junru Lu
claimed_verified
2025-11-26T09:27:16.085Z
64ddbc3f1f2dad27e1a05ac1
/avatars/4fb2753b7998c8536bfd4780d3b10a6d.svg
Junrulu
Junrulu
user
Junrulu
69269351243b2216fb75cb24
Lin Gui
69269351243b2216fb75cb25
Jiazheng Li
claimed_verified
2025-11-26T11:53:51.303Z
64be88e9af05eb17c702787c
/avatars/8953032acd739a0780e33cc46b0f9b56.svg
J Li
jiazhengli
user
jiazhengli
69269351243b2216fb75cb26
Yulan He
69269351243b2216fb75cb27
Di Yin
69269351243b2216fb75cb28
Xing Sun
claimed_verified
2025-11-27T09:59:40.138Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
The quadratic complexity of full attention limits efficient long-context processing in large language models (LLMs). Sparse attention mitigates this cost by restricting each query to attend to a subset of previous tokens; however, training-free approaches often lead to severe performance degradation. Native sparse-attention methods (e.g., NSA, MoBA) alleviate this issue, yet exhibit a critical paradox: they produce lower attention sparsity than full-attention models, despite aiming to approximate full attention, which may constrain their effectiveness. We attribute this paradox to gradient update deficiency: low-ranked key-value pairs excluded during sparse training receive neither forward contribution nor backward gradients, and thus never learn proper suppression. To overcome this limitation, we propose SSA (Sparse Sparse Attention), a unified training framework that considers both sparse and full attention and enforces bidirectional alignment at every layer. This design preserves gradient flow to all tokens while explicitly encouraging sparse-attention outputs to align with their full-attention counterparts, thereby promoting stronger sparsity. As a result, SSA achieves state-of-the-art performance under both sparse and full attention inference across multiple commonsense benchmarks. Furthermore, SSA enables models to adapt smoothly to varying sparsity budgets; performance improves consistently as more tokens are allowed to attend, supporting flexible compute-performance trade-offs at inference time. Finally, we show that native sparse-attention training surprisingly improves long-context extrapolation by mitigating the over-allocation of attention values in sink areas, with SSA demonstrating the strongest extrapolation capability.
SSA, a unified training framework for sparse attention in LLMs, achieves state-of-the-art performance by aligning sparse attention with full attention, improving long-context processing and extrapolation.
full attention sparse attention native sparse-attention methods NSA MoBA gradient update deficiency SSA Sparse Sparse Attention bidirectional alignment long-context extrapolation sink areas