SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
Many training-free sparse attention methods are effective for accelerating diffusion models. Recently, several works suggest that making sparse attention trainable can further increase sparsity while preserving generation quality. We study three key questions: (1) when do the two common masking rules, i.e., Top-k and Top-p, fail, and how can we avoid these failures? (2) why can trainable sparse attention reach higher sparsity than training-free methods? (3) what are the limitations of fine-tuning sparse attention using the diffusion loss, and how can we address them? Based on this analysis, we propose SpargeAttention2, a trainable sparse attention method that achieves high sparsity without degrading generation quality. SpargeAttention2 includes (i) a hybrid masking rule that combines Top-k and Top-p for more robust masking at high sparsity, (ii) an efficient trainable sparse attention implementation, and (iii) a distillation-inspired fine-tuning objective to better preserve generation quality during fine-tuning using sparse attention. Experiments on video diffusion models show that SpargeAttention2 reaches 95% attention sparsity and a 16.2x attention speedup while maintaining generation quality, consistently outperforming prior sparse attention methods.
6997d8487a658569d5a101d2
Jintao Zhang
claimed_verified
2026-02-20T08:32:47.646Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
jt-zhang
user
jt-zhang
6997d8487a658569d5a101d3
Kai Jiang
6997d8487a658569d5a101d4
Chendong Xiang
admin_assigned
2026-02-20T09:02:35.487Z
6329bdbbde087eac2921e6a9
https://cdn-avatars.huggingface.co/v1/production/uploads/1663679904323-noauth.jpeg
Xiangchendong
Xiang-cd
user
Xiang-cd
6997d8487a658569d5a101d5
Weiqi Feng
admin_assigned
2026-02-20T09:02:42.013Z
6698a1efbf3edea3110a6e66
https://cdn-avatars.huggingface.co/v1/production/uploads/6698a1efbf3edea3110a6e66/WzwIu_8NKEcN10CEw8G8O.jpeg
Weiqi Feng
Fengwq22
user
Fengwq22
6997d8487a658569d5a101d6
Yuezhou Hu
admin_assigned
2026-02-20T09:02:47.984Z
67dc66fe55c24fc4f981a4ab
/avatars/7bd900ade802d99db7c562ad6c2f6661.svg
Yuezhou Hu
yuezhouhu
user
yuezhouhu
6997d8487a658569d5a101d7
Haocheng Xi
6997d8487a658569d5a101d8
Jianfei Chen
6997d8487a658569d5a101d9
Jun Zhu
SpargeAttention2: Trainable Sparse Attention via Hybrid Top-k+Top-p Masking and Distillation Fine-Tuning
Many training-free sparse attention methods are effective for accelerating diffusion models. Recently, several works suggest that making sparse attention trainable can further increase sparsity while preserving generation quality. We study three key questions: (1) when do the two common masking rules, i.e., Top-k and Top-p, fail, and how can we avoid these failures? (2) why can trainable sparse attention reach higher sparsity than training-free methods? (3) what are the limitations of fine-tuning sparse attention using the diffusion loss, and how can we address them? Based on this analysis, we propose SpargeAttention2, a trainable sparse attention method that achieves high sparsity without degrading generation quality. SpargeAttention2 includes (i) a hybrid masking rule that combines Top-k and Top-p for more robust masking at high sparsity, (ii) an efficient trainable sparse attention implementation, and (iii) a distillation-inspired fine-tuning objective to better preserve generation quality during fine-tuning using sparse attention. Experiments on video diffusion models show that SpargeAttention2 reaches 95% attention sparsity and a 16.2x attention speedup while maintaining generation quality, consistently outperforming prior sparse attention methods.
A trainable sparse attention method called SpargeAttention2 is proposed that achieves high sparsity in diffusion models while maintaining generation quality through hybrid masking rules and distillation-inspired fine-tuning.
sparse attention diffusion models Top-k Top-p trainable sparse attention hybrid masking rule distillation-inspired fine-tuning attention sparsity attention speedup