OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at sim25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.Project Page: https://omniforcing.com{https://omniforcing.com}
69b6ae1c91a28162b233faf0
Yaofeng Su
claimed_verified
2026-03-16T10:40:50.576Z
646eac510867c99c2d3fde08
https://cdn-avatars.huggingface.co/v1/production/uploads/646eac510867c99c2d3fde08/fvIiW7zj4aTNbp16kTNBA.jpeg
Yaofeng Su
Exploration
user
Exploration
69b6ae1c91a28162b233faf1
Yuming Li
69b6ae1c91a28162b233faf2
Zeyue Xue
69b6ae1c91a28162b233faf3
Jie Huang
69b6ae1c91a28162b233faf4
Siming Fu
69b6ae1c91a28162b233faf5
Haoran Li
69b6ae1c91a28162b233faf6
Ying Li
69b6ae1c91a28162b233faf7
Zezhong Qian
admin_assigned
2026-03-16T10:57:36.640Z
67852e5a3d49517c54365945
/avatars/3ab8df5628e91bf89ac39f6fd6955c3e.svg
Zezhong Qian
XuWuLingYu
user
XuWuLingYu
69b6ae1c91a28162b233faf8
Haoyang Huang
69b6ae1c91a28162b233faf9
Nan Duan
OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at sim25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.Project Page: https://omniforcing.com{https://omniforcing.com}
OmniForcing distills a dual-stream bidirectional diffusion model into a streaming autoregressive generator while addressing training instability and synchronization issues through asymmetric alignment and specialized token mechanisms.
diffusion models bidirectional attention causal distillation dual-stream architecture audio token sparsity cross-modal synchronization self-forcing distillation rolling KV-cache audio sink token Identity RoPE constraint