SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer
Diffusion Transformers have recently demonstrated remarkable performance in video generation. However, the long input sequences result in high computational latency due to the quadratic complexity of full attention. Various sparse attention mechanisms have been proposed. Training-free sparse attention is constrained by limited sparsity and thus offers modest acceleration, whereas training-based methods can reach much higher sparsity but demand substantial data and computation for training. In this work, we propose SALAD, introducing a lightweight linear attention branch in parallel with the sparse attention. By incorporating an input-dependent gating mechanism to finely balance the two branches, our method attains 90% sparsity and 1.72x inference speedup, while maintaining generation quality comparable to the full attention baseline. Moreover, our finetuning process is highly efficient, requiring only 2,000 video samples and 1,600 training steps with a batch size of 8.
6976f03f5d41524304c13658
Tongcheng Fang
claimed_verified
2026-01-26T08:28:39.635Z
6524d2dae5d47a2f52ce0b03
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/KjA0bpIAXQHqtW_WEcEb8.jpeg
Tongcheng Fang
Stein-Fun
user
Stein-Fun
6976f03f5d41524304c13659
Hanling Zhang
6976f03f5d41524304c1365a
Ruiqi Xie
claimed_verified
2026-01-26T08:28:37.024Z
66f6442e04f2d5ae9789ab2b
/avatars/72beedaeda45311725aabe80a8ec91a5.svg
Ricky Xie
RickyHsieh
user
RickyHsieh
6976f03f5d41524304c1365b
Zhuo Han
6976f03f5d41524304c1365c
Xin Tao
6976f03f5d41524304c1365d
Tianchen Zhao
6976f03f5d41524304c1365e
Pengfei Wan
6976f03f5d41524304c1365f
Wenbo Ding
6976f03f5d41524304c13660
Wanli Ouyang
6976f03f5d41524304c13661
Xuefei Ning
6976f03f5d41524304c13662
Yu Wang
SALAD: Achieve High-Sparsity Attention via Efficient Linear Attention Tuning for Video Diffusion Transformer
Diffusion Transformers have recently demonstrated remarkable performance in video generation. However, the long input sequences result in high computational latency due to the quadratic complexity of full attention. Various sparse attention mechanisms have been proposed. Training-free sparse attention is constrained by limited sparsity and thus offers modest acceleration, whereas training-based methods can reach much higher sparsity but demand substantial data and computation for training. In this work, we propose SALAD, introducing a lightweight linear attention branch in parallel with the sparse attention. By incorporating an input-dependent gating mechanism to finely balance the two branches, our method attains 90% sparsity and 1.72x inference speedup, while maintaining generation quality comparable to the full attention baseline. Moreover, our finetuning process is highly efficient, requiring only 2,000 video samples and 1,600 training steps with a batch size of 8.
Diffusion Transformers for video generation are enhanced with SALAD, a method that combines linear and sparse attention branches to achieve high sparsity and speedup while maintaining quality and requiring minimal training data.
diffusion transformers video generation sparse attention linear attention input-dependent gating mechanism inference speedup parameter-efficient fine-tuning