Efficient Autoregressive Video Diffusion with Dummy Head
The autoregressive video diffusion model has recently gained considerable research interest due to its causal modeling and iterative denoising. In this work, we identify that the multi-head self-attention in these models under-utilizes historical frames: approximately 25% heads attend almost exclusively to the current frame, and discarding their KV caches incurs only minor performance degradation. Building upon this, we propose Dummy Forcing, a simple yet effective method to control context accessibility across different heads. Specifically, the proposed heterogeneous memory allocation reduces head-wise context redundancy, accompanied by dynamic head programming to adaptively classify head types. Moreover, we develop a context packing technique to achieve more aggressive cache compression. Without additional training, our Dummy Forcing delivers up to 2.0x speedup over the baseline, supporting video generation at 24.3 FPS with less than 0.5% quality drop. Project page is available at https://csguoh.github.io/project/DummyForcing/.
69844621e34659da7e1f50d5
Hang Guo
claimed_verified
2026-02-05T10:52:07.030Z
67d6bb22eab66ce9cb4e3662
/avatars/814704eef9e6907d9d4ab407b605566b.svg
Hang Guo
HangGuo
user
HangGuo
69844621e34659da7e1f50d6
Zhaoyang Jia
69844621e34659da7e1f50d7
Jiahao Li
69844621e34659da7e1f50d8
Bin Li
69844621e34659da7e1f50d9
Yuanhao Cai
69844621e34659da7e1f50da
Jiangshan Wang
69844621e34659da7e1f50db
Yawei Li
69844621e34659da7e1f50dc
Yan Lu
Efficient Autoregressive Video Diffusion with Dummy Head
The autoregressive video diffusion model has recently gained considerable research interest due to its causal modeling and iterative denoising. In this work, we identify that the multi-head self-attention in these models under-utilizes historical frames: approximately 25% heads attend almost exclusively to the current frame, and discarding their KV caches incurs only minor performance degradation. Building upon this, we propose Dummy Forcing, a simple yet effective method to control context accessibility across different heads. Specifically, the proposed heterogeneous memory allocation reduces head-wise context redundancy, accompanied by dynamic head programming to adaptively classify head types. Moreover, we develop a context packing technique to achieve more aggressive cache compression. Without additional training, our Dummy Forcing delivers up to 2.0x speedup over the baseline, supporting video generation at 24.3 FPS with less than 0.5% quality drop. Project page is available at https://csguoh.github.io/project/DummyForcing/.
Autoregressive video diffusion models suffer from inefficient attention mechanisms that underutilize historical frames, but a new method called Dummy Forcing improves efficiency through heterogeneous memory allocation and dynamic head programming while maintaining quality.
autoregressive video diffusion model multi-head self-attention causal modeling iterative denoising KV caches heterogeneous memory allocation dynamic head programming context packing cache compression