EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents
Human behaviors in the real world naturally encode rich, long-term contextual information that can be leveraged to train embodied agents for perception, understanding, and acting. However, existing capture systems typically rely on costly studio setups and wearable devices, limiting the large-scale collection of scene-conditioned human motion data in the wild. To address this, we propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones. Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes within a unified metric world coordinate frame. The proposed method allows metric-scale and scene-consistent capture in everyday environments without static cameras or markers, bridging human motion and scene geometry seamlessly. Compared with optical capture ground truth, we demonstrate that the dual-view setting exhibits a remarkable ability to mitigate depth ambiguity, achieving superior alignment and reconstruction performance over single iphone or monocular models. Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune on feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via sim-to-real RL to replicate human motions depicted in videos. Experimental results validate the effectiveness of our pipeline and its contributions towards advancing embodied AI research.
69a16d76a13deaa449448acd
Wenjia Wang
claimed_verified
2026-02-27T16:39:01.959Z
6437a813fac5ea753f1c72d2
/avatars/69e60e60497e404149a1dad46649dad4.svg
wenjia Wang
WenjiaWang
user
WenjiaWang
69a16d76a13deaa449448ace
Liang Pan
69a16d76a13deaa449448acf
Huaijin Pi
69a16d76a13deaa449448ad0
Yuke Lou
69a16d76a13deaa449448ad1
Xuqian Ren
69a16d76a13deaa449448ad2
Yifan Wu
69a16d76a13deaa449448ad3
Zhouyingcheng Liao
69a16d76a13deaa449448ad4
Lei Yang
69a16d76a13deaa449448ad5
Rishabh Dabral
69a16d76a13deaa449448ad6
Christian Theobalt
69a16d76a13deaa449448ad7
Taku Komura
EmbodMocap: In-the-Wild 4D Human-Scene Reconstruction for Embodied Agents
Human behaviors in the real world naturally encode rich, long-term contextual information that can be leveraged to train embodied agents for perception, understanding, and acting. However, existing capture systems typically rely on costly studio setups and wearable devices, limiting the large-scale collection of scene-conditioned human motion data in the wild. To address this, we propose EmbodMocap, a portable and affordable data collection pipeline using two moving iPhones. Our key idea is to jointly calibrate dual RGB-D sequences to reconstruct both humans and scenes within a unified metric world coordinate frame. The proposed method allows metric-scale and scene-consistent capture in everyday environments without static cameras or markers, bridging human motion and scene geometry seamlessly. Compared with optical capture ground truth, we demonstrate that the dual-view setting exhibits a remarkable ability to mitigate depth ambiguity, achieving superior alignment and reconstruction performance over single iphone or monocular models. Based on the collected data, we empower three embodied AI tasks: monocular human-scene-reconstruction, where we fine-tune on feedforward models that output metric-scale, world-space aligned humans and scenes; physics-based character animation, where we prove our data could be used to scale human-object interaction skills and scene-aware motion tracking; and robot motion control, where we train a humanoid robot via sim-to-real RL to replicate human motions depicted in videos. Experimental results validate the effectiveness of our pipeline and its contributions towards advancing embodied AI research.
A portable dual-iPhone system enables metric-scale human-scene reconstruction and supports embodied AI tasks including physics-based animation and robot motion control.
embodied agents RGB-D sequences metric world coordinate frame monocular human-scene-reconstruction physics-based character animation sim-to-real RL humanoid robot