HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions
We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and monocular videos. Existing methods suffer from a perception-simulation gap: visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization, which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench, a new benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world humanoid robots.
69b8db6d6b3f36c249fbc4d6
Yukang Cao
admin_assigned
2026-03-17T08:40:28.207Z
63a07c3ab5515dccd40fdb71
https://cdn-avatars.huggingface.co/v1/production/uploads/63a07c3ab5515dccd40fdb71/ly3pwhjWVge25LAeVgriV.png
Yukang Cao
yukangcao
user
yukangcao
69b8db6d6b3f36c249fbc4d7
Haozhe Xie
claimed_verified
2026-03-17T08:19:14.400Z
63f47b5321eb234ab739e91a
https://cdn-avatars.huggingface.co/v1/production/uploads/63f47b5321eb234ab739e91a/vWfFNVtMkHl8gieha5PPd.jpeg
Haozhe Xie
hzxie
user
hzxie
69b8db6d6b3f36c249fbc4d8
Fangzhou Hong
69b8db6d6b3f36c249fbc4d9
Long Zhuo
69b8db6d6b3f36c249fbc4da
Zhaoxi Chen
admin_assigned
2026-03-17T08:40:38.997Z
62fc8cf7ee999004b5a8b982
/avatars/6c5dda9e58747054a989f077a078f3dc.svg
Zhaoxi Chen
FrozenBurning
user
FrozenBurning
69b8db6d6b3f36c249fbc4db
Liang Pan
69b8db6d6b3f36c249fbc4dc
Ziwei Liu
admin_assigned
2026-03-17T08:40:55.176Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
liuziwei7
user
liuziwei7
HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactions
We present HSImul3R, a unified framework for simulation-ready 3D reconstruction of human-scene interactions (HSI) from casual captures, including sparse-view images and monocular videos. Existing methods suffer from a perception-simulation gap: visually plausible reconstructions often violate physical constraints, leading to instability in physics engines and failure in embodied AI applications. To bridge this gap, we introduce a physically-grounded bi-directional optimization pipeline that treats the physics simulator as an active supervisor to jointly refine human dynamics and scene geometry. In the forward direction, we employ Scene-targeted Reinforcement Learning to optimize human motion under dual supervision of motion fidelity and contact stability. In the reverse direction, we propose Direct Simulation Reward Optimization, which leverages simulation feedback on gravitational stability and interaction success to refine scene geometry. We further present HSIBench, a new benchmark with diverse objects and interaction scenarios. Extensive experiments demonstrate that HSImul3R produces the first stable, simulation-ready HSI reconstructions and can be directly deployed to real-world humanoid robots.
HSImul3R presents a unified framework for 3D reconstruction of human-scene interactions that bridges the perception-simulation gap through physics-grounded bidirectional optimization and reinforcement learning.
3D reconstruction human-scene interactions physics engine bi-directional optimization reinforcement learning simulation feedback gravitational stability contact stability simulation-ready embodied AI humanoid robots