SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable
  Sparse-Linear Attention
In Diffusion Transformer (DiT) models, particularly for video generation,
attention latency is a major bottleneck due to the long sequence length and the
quadratic complexity. We find that attention weights can be separated into two
parts: a small fraction of large weights with high rank and the remaining
weights with very low rank. This naturally suggests applying sparse
acceleration to the first part and low-rank acceleration to the second. Based
on this finding, we propose SLA (Sparse-Linear Attention), a trainable
attention method that fuses sparse and linear attention to accelerate diffusion
models. SLA classifies attention weights into critical, marginal, and
negligible categories, applying O(N^2) attention to critical weights, O(N)
attention to marginal weights, and skipping negligible ones. SLA combines these
computations into a single GPU kernel and supports both forward and backward
passes. With only a few fine-tuning steps using SLA, DiT models achieve a 20x
reduction in attention computation, resulting in significant acceleration
without loss of generation quality. Experiments show that SLA reduces attention
computation by 95% without degrading end-to-end generation quality,
outperforming baseline methods. In addition, we implement an efficient GPU
kernel for SLA, which yields a 13.7x speedup in attention computation and a
2.2x end-to-end speedup in video generation on Wan2.1-1.3B.
68db424ed2bf1f4b15ec730a
Jintao Zhang
claimed_verified
2025-09-30T06:32:43.935Z
66c0a08bac74db25de8427ec
https://cdn-avatars.huggingface.co/v1/production/uploads/66c0a08bac74db25de8427ec/9D-piDBZqSt6KNkHImmkv.jpeg
Jintao Zhang
jt-zhang
user
jt-zhang
68db424ed2bf1f4b15ec730b
Haoxu Wang
claimed_verified
2025-10-08T08:03:49.676Z
658c1802a1105f8157ad1db9
https://cdn-avatars.huggingface.co/v1/production/uploads/658c1802a1105f8157ad1db9/WzjY29SkngxkKfiTYcssh.jpeg
whx1003
whx1003
user
whx1003
68db424ed2bf1f4b15ec730c
Kai Jiang
68db424ed2bf1f4b15ec730d
Shuo Yang
claimed_verified
2026-03-11T08:47:22.963Z
642b970ceb31218a5f204a29
/avatars/582287f477bbb1a0842787145e375fd3.svg
andy-yang
andy-yang
user
andy-yang
68db424ed2bf1f4b15ec730e
Kaiwen Zheng
claimed_verified
2025-10-18T04:50:58.075Z
652bf7edc3cba555d5673c6e
/avatars/78f6416c30203b30671f8423f061c657.svg
Kaiwen Zheng
worstcoder
user
worstcoder
68db424ed2bf1f4b15ec730f
Haocheng Xi
68db424ed2bf1f4b15ec7310
Ziteng Wang
68db424ed2bf1f4b15ec7311
Hongzhou Zhu
claimed_verified
2025-11-26T09:28:42.361Z
64c269a52d73768f07ac266c
/avatars/d497a960f8aef6a974907b68ed750c1c.svg
Zhu Hongzhou
zhuhz22
user
zhuhz22
68db424ed2bf1f4b15ec7312
Min Zhao
68db424ed2bf1f4b15ec7313
Ion Stoica
68db424ed2bf1f4b15ec7314
Joseph E. Gonzalez
68db424ed2bf1f4b15ec7315
Jun Zhu
68db424ed2bf1f4b15ec7316
Jianfei Chen
SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable
  Sparse-Linear Attention
In Diffusion Transformer (DiT) models, particularly for video generation,
attention latency is a major bottleneck due to the long sequence length and the
quadratic complexity. We find that attention weights can be separated into two
parts: a small fraction of large weights with high rank and the remaining
weights with very low rank. This naturally suggests applying sparse
acceleration to the first part and low-rank acceleration to the second. Based
on this finding, we propose SLA (Sparse-Linear Attention), a trainable
attention method that fuses sparse and linear attention to accelerate diffusion
models. SLA classifies attention weights into critical, marginal, and
negligible categories, applying O(N^2) attention to critical weights, O(N)
attention to marginal weights, and skipping negligible ones. SLA combines these
computations into a single GPU kernel and supports both forward and backward
passes. With only a few fine-tuning steps using SLA, DiT models achieve a 20x
reduction in attention computation, resulting in significant acceleration
without loss of generation quality. Experiments show that SLA reduces attention
computation by 95% without degrading end-to-end generation quality,
outperforming baseline methods. In addition, we implement an efficient GPU
kernel for SLA, which yields a 13.7x speedup in attention computation and a
2.2x end-to-end speedup in video generation on Wan2.1-1.3B.
SLA, a trainable attention method combining sparse and linear attention, accelerates Diffusion Transformer models for video generation with minimal quality loss.
Diffusion Transformer attention latency sequence length quadratic complexity sparse acceleration low-rank acceleration SLA critical weights marginal weights negligible weights GPU kernel fine-tuning attention computation generation quality end-to-end speedup