H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos
Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: https://showlab.github.io/H2R-Grounder/
693b956d9874a2a5e4ffb34c
Hai Ci
admin_assigned
2025-12-12T13:09:23.967Z
647352516a972f252de1fd58
/avatars/8e93ca66e01f047c5fb28e1c4f737e8e.svg
Hai Ci
HaiCi
user
HaiCi
693b956d9874a2a5e4ffb34d
Xiaokang Liu
claimed_verified
2026-02-04T12:35:00.699Z
672d7efba3cac338ea972881
/avatars/ff6ef1c4e08ef66c893e74aec8b5750c.svg
Xiaokang Liu
hiskiv
user
hiskiv
693b956d9874a2a5e4ffb34e
Pei Yang
693b956d9874a2a5e4ffb34f
Yiren Song
admin_assigned
2025-12-12T13:09:39.096Z
64311a95034ecbefddd141ef
/avatars/b6dc5ca373bedbaa368208517954c375.svg
Yiren Song
yiren98
user
yiren98
693b956d9874a2a5e4ffb350
Mike Zheng Shou
admin_assigned
2025-12-12T13:09:33.475Z
661ab3da2b14565c7acccf5c
/avatars/fa4fc03664803e02aede4d4c3d50b393.svg
Mike Zheng Shou
AnalMom
user
AnalMom
H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos
Robots that learn manipulation skills from everyday human videos could acquire broad capabilities without tedious robot data collection. We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos with realistic, physically grounded interactions. Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale. We introduce a transferable representation that bridges the embodiment gap: by inpainting the robot arm in training videos to obtain a clean background and overlaying a simple visual cue (a marker and arrow indicating the gripper's position and orientation), we can condition a generative model to insert the robot arm back into the scene. At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions. We fine-tune a SOTA video diffusion model (Wan 2.2) in an in-context learning manner to ensure temporal coherence and leveraging of its rich prior knowledge. Empirical results demonstrate that our approach achieves significantly more realistic and grounded robot motions compared to baselines, pointing to a promising direction for scaling up robot learning from unlabeled human videos. Project page: https://showlab.github.io/H2R-Grounder/
A video-to-video translation framework converts human-object interaction videos into realistic robot manipulation videos using unpaired training data and a generative model.
video-to-video translation motion-consistent physically grounded interactions unpaired robot videos transferable representation inpainting visual cue generative model in-context learning video diffusion model temporal coherence prior knowledge