Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control
Extended reality (XR) demands generative models that respond to users' tracked real-world motion, yet current video world models accept only coarse control signals such as text or keyboard input, limiting their utility for embodied interaction. We introduce a human-centric video world model that is conditioned on both tracked head pose and joint-level hand poses. For this purpose, we evaluate existing diffusion transformer conditioning strategies and propose an effective mechanism for 3D head and hand control, enabling dexterous hand--object interactions. We train a bidirectional video diffusion model teacher using this strategy and distill it into a causal, interactive system that generates egocentric virtual environments. We evaluate this generated reality system with human subjects and demonstrate improved task performance as well as a significantly higher level of perceived amount of control over the performed actions compared with relevant baselines.
699bbf5bf723198bd53a6309
Linxi Xie
699bbf5bf723198bd53a630a
Lisong C. Sun
699bbf5bf723198bd53a630b
Ashley Neall
699bbf5bf723198bd53a630c
Tong Wu
699bbf5bf723198bd53a630d
Shengqu Cai
699bbf5bf723198bd53a630e
Gordon Wetzstein
Generated Reality: Human-centric World Simulation using Interactive Video Generation with Hand and Camera Control
Extended reality (XR) demands generative models that respond to users' tracked real-world motion, yet current video world models accept only coarse control signals such as text or keyboard input, limiting their utility for embodied interaction. We introduce a human-centric video world model that is conditioned on both tracked head pose and joint-level hand poses. For this purpose, we evaluate existing diffusion transformer conditioning strategies and propose an effective mechanism for 3D head and hand control, enabling dexterous hand--object interactions. We train a bidirectional video diffusion model teacher using this strategy and distill it into a causal, interactive system that generates egocentric virtual environments. We evaluate this generated reality system with human subjects and demonstrate improved task performance as well as a significantly higher level of perceived amount of control over the performed actions compared with relevant baselines.
A human-centric video world model conditioned on tracked head and hand poses is introduced, enabling dexterous interactions through a bidirectional video diffusion model trained for egocentric virtual environment generation.
video world models diffusion transformer 3D head pose joint-level hand poses dexterous hand-object interactions bidirectional video diffusion model causal interactive system egocentric virtual environments