MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE
We introduce MotionCrafter, a video diffusion-based framework that jointly reconstructs 4D geometry and estimates dense motion from a monocular video. The core of our method is a novel joint representation of dense 3D point maps and 3D scene flows in a shared coordinate system, and a novel 4D VAE to effectively learn this representation. Unlike prior work that forces the 3D value and latents to align strictly with RGB VAE latents-despite their fundamentally different distributions-we show that such alignment is unnecessary and leads to suboptimal performance. Instead, we introduce a new data normalization and VAE training strategy that better transfers diffusion priors and greatly improves reconstruction quality. Extensive experiments across multiple datasets demonstrate that MotionCrafter achieves state-of-the-art performance in both geometry reconstruction and dense scene flow estimation, delivering 38.64% and 25.0% improvements in geometry and motion reconstruction, respectively, all without any post-optimization. Project page: https://ruijiezhu94.github.io/MotionCrafter_Page
698ab60e1b2dc6b37d61aff8
Ruijie Zhu
claimed_verified
2026-02-10T09:06:13.882Z
6697ac8427e4e21a3a92da27
https://cdn-avatars.huggingface.co/v1/production/uploads/6697ac8427e4e21a3a92da27/9vn07-1_BBDk9zfDtDpcG.png
Ruijie Zhu
RuijieZhu
user
RuijieZhu
698ab60e1b2dc6b37d61aff9
Jiahao Lu
698ab60e1b2dc6b37d61affa
Wenbo Hu
698ab60e1b2dc6b37d61affb
Xiaoguang Han
698ab60e1b2dc6b37d61affc
Jianfei Cai
698ab60e1b2dc6b37d61affd
Ying Shan
698ab60e1b2dc6b37d61affe
Chuanxia Zheng
MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAE
We introduce MotionCrafter, a video diffusion-based framework that jointly reconstructs 4D geometry and estimates dense motion from a monocular video. The core of our method is a novel joint representation of dense 3D point maps and 3D scene flows in a shared coordinate system, and a novel 4D VAE to effectively learn this representation. Unlike prior work that forces the 3D value and latents to align strictly with RGB VAE latents-despite their fundamentally different distributions-we show that such alignment is unnecessary and leads to suboptimal performance. Instead, we introduce a new data normalization and VAE training strategy that better transfers diffusion priors and greatly improves reconstruction quality. Extensive experiments across multiple datasets demonstrate that MotionCrafter achieves state-of-the-art performance in both geometry reconstruction and dense scene flow estimation, delivering 38.64% and 25.0% improvements in geometry and motion reconstruction, respectively, all without any post-optimization. Project page: https://ruijiezhu94.github.io/MotionCrafter_Page
MotionCrafter is a video diffusion framework that jointly reconstructs 4D geometry and estimates dense motion using a novel joint representation and 4D VAE architecture.
video diffusion 4D geometry dense motion estimation 3D point maps 3D scene flows shared coordinate system 4D VAE RGB VAE latents data normalization VAE training strategy diffusion priors