SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time
We present SpaceTimePilot, a video diffusion model that disentangles space and time for controllable generative rendering. Given a monocular video, SpaceTimePilot can independently alter the camera viewpoint and the motion sequence within the generative process, re-rendering the scene for continuous and arbitrary exploration across space and time. To achieve this, we introduce an effective animation time-embedding mechanism in the diffusion process, allowing explicit control of the output video's motion sequence with respect to that of the source video. As no datasets provide paired videos of the same dynamic scene with continuous temporal variations, we propose a simple yet effective temporal-warping training scheme that repurposes existing multi-view datasets to mimic temporal differences. This strategy effectively supervises the model to learn temporal control and achieve robust space-time disentanglement. To further enhance the precision of dual control, we introduce two additional components: an improved camera-conditioning mechanism that allows altering the camera from the first frame, and CamxTime, the first synthetic space-and-time full-coverage rendering dataset that provides fully free space-time video trajectories within a scene. Joint training on the temporal-warping scheme and the CamxTime dataset yields more precise temporal control. We evaluate SpaceTimePilot on both real-world and synthetic data, demonstrating clear space-time disentanglement and strong results compared to prior work. Project page: https://zheninghuang.github.io/Space-Time-Pilot/ Code: https://github.com/ZheningHuang/spacetimepilot
6955e463832867f253525627
Zhening Huang
claimed_verified
2026-03-24T09:23:29.783Z
659f4a6d697a41751b9c192c
/avatars/82748153932584e451f2d3245c0e9043.svg
huang
zhening
user
zhening
6955e463832867f253525628
Hyeonho Jeong
claimed_verified
2026-01-05T16:03:30.501Z
656ee8008bb9f4f8d95bd8f7
/avatars/4069d70f1279d928da521211c495d638.svg
Hyeonho Jeong
hyeonho-jeong-video
user
hyeonho-jeong-video
6955e463832867f253525629
Xuelin Chen
6955e463832867f25352562a
Yulia Gryaditskaya
6955e463832867f25352562b
Tuanfeng Y. Wang
6955e463832867f25352562c
Joan Lasenby
6955e463832867f25352562d
Chun-Hao Huang
SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time
We present SpaceTimePilot, a video diffusion model that disentangles space and time for controllable generative rendering. Given a monocular video, SpaceTimePilot can independently alter the camera viewpoint and the motion sequence within the generative process, re-rendering the scene for continuous and arbitrary exploration across space and time. To achieve this, we introduce an effective animation time-embedding mechanism in the diffusion process, allowing explicit control of the output video's motion sequence with respect to that of the source video. As no datasets provide paired videos of the same dynamic scene with continuous temporal variations, we propose a simple yet effective temporal-warping training scheme that repurposes existing multi-view datasets to mimic temporal differences. This strategy effectively supervises the model to learn temporal control and achieve robust space-time disentanglement. To further enhance the precision of dual control, we introduce two additional components: an improved camera-conditioning mechanism that allows altering the camera from the first frame, and CamxTime, the first synthetic space-and-time full-coverage rendering dataset that provides fully free space-time video trajectories within a scene. Joint training on the temporal-warping scheme and the CamxTime dataset yields more precise temporal control. We evaluate SpaceTimePilot on both real-world and synthetic data, demonstrating clear space-time disentanglement and strong results compared to prior work. Project page: https://zheninghuang.github.io/Space-Time-Pilot/ Code: https://github.com/ZheningHuang/spacetimepilot
SpaceTimePilot is a video diffusion model enabling independent control of spatial viewpoint and temporal motion through a time-embedding mechanism, temporal-warping training, and a synthetic dataset for precise space-time disentanglement.
video diffusion model animation time-embedding mechanism temporal-warping training scheme camera-conditioning mechanism CamxTime dataset space-time disentanglement diffusion process multi-view datasets