Simulating the Visual World with Artificial Intelligence: A Roadmap
The landscape of video generation is shifting, from a focus on generating visually appealing clips to building virtual environments that support interaction and maintain physical plausibility. These developments point toward the emergence of video foundation models that function not only as visual generators but also as implicit world models, models that simulate the physical dynamics, agent-environment interactions, and task planning that govern real or imagined worlds. This survey provides a systematic overview of this evolution, conceptualizing modern video foundation models as the combination of two core components: an implicit world model and a video renderer. The world model encodes structured knowledge about the world, including physical laws, interaction dynamics, and agent behavior. It serves as a latent simulation engine that enables coherent visual reasoning, long-term temporal consistency, and goal-driven planning. The video renderer transforms this latent simulation into realistic visual observations, effectively producing videos as a "window" into the simulated world. We trace the progression of video generation through four generations, in which the core capabilities advance step by step, ultimately culminating in a world model, built upon a video generation model, that embodies intrinsic physical plausibility, real-time multimodal interaction, and planning capabilities spanning multiple spatiotemporal scales. For each generation, we define its core characteristics, highlight representative works, and examine their application domains such as robotics, autonomous driving, and interactive gaming. Finally, we discuss open challenges and design principles for next-generation world models, including the role of agent intelligence in shaping and evaluating these systems. An up-to-date list of related works is maintained at this link.
691708feb63bfc66e04988b1
Jingtong Yue
claimed_verified
2025-11-18T14:09:18.436Z
6641f48c453a0a94639f1a73
/avatars/ad9779057d00d0ff3c82163fc0c52a0a.svg
Jingtong Yue
little-public-up
user
little-public-up
691708feb63bfc66e04988b2
Ziqi Huang
claimed_verified
2025-11-18T14:09:20.610Z
60efe7fa0d920bc7805cada5
https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
Ziqi Huang
Ziqi
user
Ziqi
691708feb63bfc66e04988b3
Zhaoxi Chen
691708feb63bfc66e04988b4
Xintao Wang
691708feb63bfc66e04988b5
Pengfei Wan
691708feb63bfc66e04988b6
Ziwei Liu
Simulating the Visual World with Artificial Intelligence: A Roadmap
The landscape of video generation is shifting, from a focus on generating visually appealing clips to building virtual environments that support interaction and maintain physical plausibility. These developments point toward the emergence of video foundation models that function not only as visual generators but also as implicit world models, models that simulate the physical dynamics, agent-environment interactions, and task planning that govern real or imagined worlds. This survey provides a systematic overview of this evolution, conceptualizing modern video foundation models as the combination of two core components: an implicit world model and a video renderer. The world model encodes structured knowledge about the world, including physical laws, interaction dynamics, and agent behavior. It serves as a latent simulation engine that enables coherent visual reasoning, long-term temporal consistency, and goal-driven planning. The video renderer transforms this latent simulation into realistic visual observations, effectively producing videos as a "window" into the simulated world. We trace the progression of video generation through four generations, in which the core capabilities advance step by step, ultimately culminating in a world model, built upon a video generation model, that embodies intrinsic physical plausibility, real-time multimodal interaction, and planning capabilities spanning multiple spatiotemporal scales. For each generation, we define its core characteristics, highlight representative works, and examine their application domains such as robotics, autonomous driving, and interactive gaming. Finally, we discuss open challenges and design principles for next-generation world models, including the role of agent intelligence in shaping and evaluating these systems. An up-to-date list of related works is maintained at this link.
Video generation is evolving towards foundation models that integrate world simulation and rendering to produce physically plausible and interactive videos.
video foundation models implicit world models video renderer physical dynamics agent-environment interactions task planning latent simulation visual reasoning temporal consistency goal-driven planning physical plausibility multimodal interaction spatiotemporal scales agent intelligence