LoL: Longer than Longer, Scaling Video Generation to Hour
Recent research in long-form video generation has shifted from bidirectional to autoregressive models, yet these methods commonly suffer from error accumulation and a loss of long-term coherence. While attention sink frames have been introduced to mitigate this performance decay, they often induce a critical failure mode we term sink-collapse: the generated content repeatedly reverts to the sink frame, resulting in abrupt scene resets and cyclic motion patterns. Our analysis reveals that sink-collapse originates from an inherent conflict between the periodic structure of Rotary Position Embedding (RoPE) and the multi-head attention mechanisms prevalent in current generative models. To address it, we propose a lightweight, training-free approach that effectively suppresses this behavior by introducing multi-head RoPE jitter that breaks inter-head attention homogenization and mitigates long-horizon collapse. Extensive experiments show that our method successfully alleviates sink-collapse while preserving generation quality. To the best of our knowledge, this work achieves the first demonstration of real-time, streaming, and infinite-length video generation with little quality decay. As an illustration of this robustness, we generate continuous videos up to 12 hours in length, which, to our knowledge, is among the longest publicly demonstrated results in streaming video generation.
697cf81a6676f93322706163
Justin Cui
697cf81a6676f93322706164
Jie Wu
697cf81a6676f93322706165
Ming Li
697cf81a6676f93322706166
Tao Yang
697cf81a6676f93322706167
Xiaojie Li
697cf81a6676f93322706168
Rui Wang
697cf81a6676f93322706169
Andrew Bai
697cf81a6676f9332270616a
Yuanhao Ban
697cf81a6676f9332270616b
Cho-Jui Hsieh
LoL: Longer than Longer, Scaling Video Generation to Hour
Recent research in long-form video generation has shifted from bidirectional to autoregressive models, yet these methods commonly suffer from error accumulation and a loss of long-term coherence. While attention sink frames have been introduced to mitigate this performance decay, they often induce a critical failure mode we term sink-collapse: the generated content repeatedly reverts to the sink frame, resulting in abrupt scene resets and cyclic motion patterns. Our analysis reveals that sink-collapse originates from an inherent conflict between the periodic structure of Rotary Position Embedding (RoPE) and the multi-head attention mechanisms prevalent in current generative models. To address it, we propose a lightweight, training-free approach that effectively suppresses this behavior by introducing multi-head RoPE jitter that breaks inter-head attention homogenization and mitigates long-horizon collapse. Extensive experiments show that our method successfully alleviates sink-collapse while preserving generation quality. To the best of our knowledge, this work achieves the first demonstration of real-time, streaming, and infinite-length video generation with little quality decay. As an illustration of this robustness, we generate continuous videos up to 12 hours in length, which, to our knowledge, is among the longest publicly demonstrated results in streaming video generation.
Researchers developed a method to overcome sink-collapse in autoregressive video generation by addressing the conflict between Rotary Position Embedding and multi-head attention mechanisms, enabling real-time streaming of videos up to 12 hours long.
Rotary Position Embedding multi-head attention autoregressive models sink-collapse attention sink frames RoPE jitter video generation streaming video generation