Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
With the advancement of interactive video generation, diffusion models have increasingly demonstrated their potential as world models. However, existing approaches still struggle to simultaneously achieve memory-enabled long-term temporal consistency and high-resolution real-time generation, limiting their applicability in real-world scenarios. To address this, we present Matrix-Game 3.0, a memory-augmented interactive world model designed for 720p real-time longform video generation. Building upon Matrix-Game 2.0, we introduce systematic improvements across data, model, and inference. First, we develop an upgraded industrial-scale infinite data engine that integrates Unreal Engine-based synthetic data, large-scale automated collection from AAA games, and real-world video augmentation to produce high-quality Video-Pose-Action-Prompt quadruplet data at scale. Second, we propose a training framework for long-horizon consistency: by modeling prediction residuals and re-injecting imperfect generated frames during training, the base model learns self-correction; meanwhile, camera-aware memory retrieval and injection enable the base model to achieve long horizon spatiotemporal consistency. Third, we design a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization and VAE decoder pruning, to achieve efficient real-time inference. Experimental results show that Matrix-Game 3.0 achieves up to 40 FPS real-time generation at 720p resolution with a 5B model, while maintaining stable memory consistency over minute-long sequences. Scaling up to a 2x14B model further improves generation quality, dynamics, and generalization. Our approach provides a practical pathway toward industrial-scale deployable world models.
69dc4c9f2b6b8dc66ffeeb55
Zile Wang
69dc4c9f2b6b8dc66ffeeb56
Zexiang Liu
69dc4c9f2b6b8dc66ffeeb57
Jaixing Li
69dc4c9f2b6b8dc66ffeeb58
Kaichen Huang
69dc4c9f2b6b8dc66ffeeb59
Baixin Xu
69dc4c9f2b6b8dc66ffeeb5a
Fei Kang
claimed_verified
2026-04-13T07:32:35.288Z
67a9b36a2fbf63093c19d3de
https://cdn-avatars.huggingface.co/v1/production/uploads/67a9b36a2fbf63093c19d3de/LbB2XH0ezcJ_tdu4hU0Of.png
康飞
kangfei
user
kangfei
69dc4c9f2b6b8dc66ffeeb5b
Mengyin An
claimed_verified
2026-04-13T07:32:38.075Z
67a9664af5f1253c64259c50
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Jk9W1C8704pqxNOsqZ0d9.png
an
dearamy
user
dearamy
69dc4c9f2b6b8dc66ffeeb5c
Peiyu Wang
69dc4c9f2b6b8dc66ffeeb5d
Biao Jiang
69dc4c9f2b6b8dc66ffeeb5e
Yichen Wei
69dc4c9f2b6b8dc66ffeeb5f
Yidan Xietian
69dc4c9f2b6b8dc66ffeeb60
Jiangbo Pei
69dc4c9f2b6b8dc66ffeeb61
Liang Hu
69dc4c9f2b6b8dc66ffeeb62
Boyi Jiang
claimed_verified
2026-04-13T07:32:40.376Z
64f21a1fbb97012498ea6bca
/avatars/d17cc9ddc9054bca283fd45dca8e63ce.svg
Boyi Jiang
jiangby1993
user
jiangby1993
69dc4c9f2b6b8dc66ffeeb63
Hua Xue
69dc4c9f2b6b8dc66ffeeb64
Zidong Wang
69dc4c9f2b6b8dc66ffeeb65
Haofeng Sun
69dc4c9f2b6b8dc66ffeeb66
Wei Li
69dc4c9f2b6b8dc66ffeeb67
Wanli Ouyang
69dc4c9f2b6b8dc66ffeeb68
Xianglong He
69dc4c9f2b6b8dc66ffeeb69
Yang Liu
69dc4c9f2b6b8dc66ffeeb6a
Yangguang Li
69dc4c9f2b6b8dc66ffeeb6b
Yahui Zhou
Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
With the advancement of interactive video generation, diffusion models have increasingly demonstrated their potential as world models. However, existing approaches still struggle to simultaneously achieve memory-enabled long-term temporal consistency and high-resolution real-time generation, limiting their applicability in real-world scenarios. To address this, we present Matrix-Game 3.0, a memory-augmented interactive world model designed for 720p real-time longform video generation. Building upon Matrix-Game 2.0, we introduce systematic improvements across data, model, and inference. First, we develop an upgraded industrial-scale infinite data engine that integrates Unreal Engine-based synthetic data, large-scale automated collection from AAA games, and real-world video augmentation to produce high-quality Video-Pose-Action-Prompt quadruplet data at scale. Second, we propose a training framework for long-horizon consistency: by modeling prediction residuals and re-injecting imperfect generated frames during training, the base model learns self-correction; meanwhile, camera-aware memory retrieval and injection enable the base model to achieve long horizon spatiotemporal consistency. Third, we design a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization and VAE decoder pruning, to achieve efficient real-time inference. Experimental results show that Matrix-Game 3.0 achieves up to 40 FPS real-time generation at 720p resolution with a 5B model, while maintaining stable memory consistency over minute-long sequences. Scaling up to a 2x14B model further improves generation quality, dynamics, and generalization. Our approach provides a practical pathway toward industrial-scale deployable world models.
Matrix-Game 3.0 enhances interactive video generation through memory-augmented diffusion models that achieve real-time 720p video synthesis with long-term temporal consistency.
diffusion models world models real-time generation long-term temporal consistency Video-Pose-Action-Prompt prediction residuals camera-aware memory retrieval autoregressive distillation Distribution Matching Distillation model quantization VAE decoder pruning