CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives
We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human-scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-ready geometry by fitting planar primitives to a point cloud reconstruction of the scene, via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we make use of human-scene contact modeling (e.g., we use human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically-plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion tracking failure rates from 55.2\% to 6.9\% on human-centric video benchmarks (EMDB, PROX), while delivering a 43\% faster RL simulation throughput. We further validate it on in-the-wild videos including casually-captured videos, Internet videos, and even Sora-generated videos. This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale, greatly advancing real-to-sim applications for robotics and AR/VR.
6942d207fc8d92509a292dad
Zihan Wang
claimed_verified
2025-12-17T16:18:57.682Z
68ff166378861d3506b44e54
/avatars/16d2c04e389e9c67ad5df10f5021fb87.svg
Zihan Wang
z1hanw
user
z1hanw
6942d207fc8d92509a292dae
Jiashun Wang
6942d207fc8d92509a292daf
Jeff Tan
claimed_verified
2026-02-17T15:49:26.730Z
6616db12f60442df33b7035c
/avatars/e0737dc8cff7ddb54ebb65447da4783b.svg
Jeff Tan
jefftan969
user
jefftan969
6942d207fc8d92509a292db0
Yiwen Zhao
6942d207fc8d92509a292db1
Jessica Hodgins
6942d207fc8d92509a292db2
Shubham Tulsiani
6942d207fc8d92509a292db3
Deva Ramanan
CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives
We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video. Prior work on joint human-scene reconstruction relies on data-driven priors and joint optimization with no physics in the loop, or recovers noisy geometry with artifacts that cause motion tracking policies with scene interactions to fail. In contrast, our key insight is to recover convex, clean, and simulation-ready geometry by fitting planar primitives to a point cloud reconstruction of the scene, via a simple clustering pipeline over depth, normals, and flow. To reconstruct scene geometry that might be occluded during interactions, we make use of human-scene contact modeling (e.g., we use human posture to reconstruct the occluded seat of a chair). Finally, we ensure that human and scene reconstructions are physically-plausible by using them to drive a humanoid controller via reinforcement learning. Our approach reduces motion tracking failure rates from 55.2\% to 6.9\% on human-centric video benchmarks (EMDB, PROX), while delivering a 43\% faster RL simulation throughput. We further validate it on in-the-wild videos including casually-captured videos, Internet videos, and even Sora-generated videos. This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale, greatly advancing real-to-sim applications for robotics and AR/VR.
CRISP recovers simulation-ready human motion and scene geometry from monocular video using planar primitive fitting and reinforcement learning, significantly reducing motion tracking failures and improving simulation throughput.
human-scene reconstruction data-driven priors joint optimization convex geometry planar primitives point cloud reconstruction depth normals flow human-scene contact modeling humanoid controller reinforcement learning EMDB PROX physically-valid human motion real-to-sim applications robotics AR/VR