SLA2: Sparse-Linear Attention with Learnable Routing and QAT
Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality.
69967cd61268a6b79e0d02d9
Jintao Zhang
claimed_verified
2026-02-19T09:51:51.054Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
jt-zhang
user
jt-zhang
69967cd61268a6b79e0d02da
Haoxu Wang
claimed_verified
2026-02-19T09:51:48.596Z
658c1802a1105f8157ad1db9
https://cdn-avatars.huggingface.co/v1/production/uploads/658c1802a1105f8157ad1db9/WzjY29SkngxkKfiTYcssh.jpeg
whx1003
whx1003
user
whx1003
69967cd61268a6b79e0d02db
Kai Jiang
69967cd61268a6b79e0d02dc
Kaiwen Zheng
69967cd61268a6b79e0d02dd
Youhe Jiang
69967cd61268a6b79e0d02de
Ion Stoica
69967cd61268a6b79e0d02df
Jianfei Chen
69967cd61268a6b79e0d02e0
Jun Zhu
69967cd61268a6b79e0d02e1
Joseph E. Gonzalez
SLA2: Sparse-Linear Attention with Learnable Routing and QAT
Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality.
SLA2 improves sparse-linear attention in diffusion models by introducing a learnable router, direct attention formulation, and quantization-aware fine-tuning for enhanced efficiency and quality.
sparse-linear attention diffusion models attention sparsity learnable router quantization-aware fine-tuning attention error direct decomposition