SyncMV4D: Synchronized Multi-view Joint Diffusion of Appearance and Motion for Hand-Object Interaction Synthesis
Hand-Object Interaction (HOI) generation plays a critical role in advancing applications across animation and robotics. Current video-based methods are predominantly single-view, which impedes comprehensive 3D geometry perception and often results in geometric distortions or unrealistic motion patterns. While 3D HOI approaches can generate dynamically plausible motions, their dependence on high-quality 3D data captured in controlled laboratory settings severely limits their generalization to real-world scenarios. To overcome these limitations, we introduce SyncMV4D, the first model that jointly generates synchronized multi-view HOI videos and 4D motions by unifying visual prior, motion dynamics, and multi-view geometry. Our framework features two core innovations: (1) a Multi-view Joint Diffusion (MJD) model that co-generates HOI videos and intermediate motions, and (2) a Diffusion Points Aligner (DPA) that refines the coarse intermediate motion into globally aligned 4D metric point tracks. To tightly couple 2D appearance with 4D dynamics, we establish a closed-loop, mutually enhancing cycle. During the diffusion denoising process, the generated video conditions the refinement of the 4D motion, while the aligned 4D point tracks are reprojected to guide next-step joint generation. Experimentally, our method demonstrates superior performance to state-of-the-art alternatives in visual realism, motion plausibility, and multi-view consistency.
6925284016eb3a9f131039a9
Lingwei Dang
claimed_verified
2025-11-25T09:05:15.579Z
62ebd791fee90fca4742ead8
https://cdn-avatars.huggingface.co/v1/production/uploads/62ebd791fee90fca4742ead8/8-iJYS9Wk41l6JTGtJrNf.jpeg
levon dang
levondang
user
levondang
6925284016eb3a9f131039aa
Zonghan Li
6925284016eb3a9f131039ab
Juntong Li
admin_assigned
2025-11-25T09:36:22.719Z
6654763c965ea394ee475878
https://cdn-avatars.huggingface.co/v1/production/uploads/6654763c965ea394ee475878/jInS2XeZRObRLfyJcM1nS.jpeg
Juntong Li
qzfm
user
qzfm
6925284016eb3a9f131039ac
Hongwen Zhang
6925284016eb3a9f131039ad
Liang An
6925284016eb3a9f131039ae
Yebin Liu
admin_assigned
2025-11-25T09:36:04.701Z
62e14dbe4db2175cd2735a80
/avatars/e6385dcedcb97e1b36281b49210321aa.svg
Yebin Liu
YebinLiu
user
YebinLiu
6925284016eb3a9f131039af
Qingyao Wu
SyncMV4D: Synchronized Multi-view Joint Diffusion of Appearance and Motion for Hand-Object Interaction Synthesis
Hand-Object Interaction (HOI) generation plays a critical role in advancing applications across animation and robotics. Current video-based methods are predominantly single-view, which impedes comprehensive 3D geometry perception and often results in geometric distortions or unrealistic motion patterns. While 3D HOI approaches can generate dynamically plausible motions, their dependence on high-quality 3D data captured in controlled laboratory settings severely limits their generalization to real-world scenarios. To overcome these limitations, we introduce SyncMV4D, the first model that jointly generates synchronized multi-view HOI videos and 4D motions by unifying visual prior, motion dynamics, and multi-view geometry. Our framework features two core innovations: (1) a Multi-view Joint Diffusion (MJD) model that co-generates HOI videos and intermediate motions, and (2) a Diffusion Points Aligner (DPA) that refines the coarse intermediate motion into globally aligned 4D metric point tracks. To tightly couple 2D appearance with 4D dynamics, we establish a closed-loop, mutually enhancing cycle. During the diffusion denoising process, the generated video conditions the refinement of the 4D motion, while the aligned 4D point tracks are reprojected to guide next-step joint generation. Experimentally, our method demonstrates superior performance to state-of-the-art alternatives in visual realism, motion plausibility, and multi-view consistency.
SyncMV4D generates realistic and consistent multi-view 3D Hand-Object Interaction videos and 4D motions by integrating visual priors, motion dynamics, and multi-view geometry.
Hand-Object Interaction HOI SyncMV4D Multi-view Joint Diffusion MJD Diffusion Points Aligner DPA diffusion denoising 4D motions multi-view consistency visual realism motion plausibility