OneHOI: Unifying Human-Object Interaction Generation and Editing
Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.
69e1d779860e47828575b5b4
Jiun Tian Hoe
claimed_verified
2026-04-17T06:58:52.591Z
6423d18e2a1f1b9ecd5a1538
/avatars/7d4b96b21e39126ad75b5bc944f74d6b.svg
Hoe Jiun Tian
jiuntian
user
jiuntian
69e1d779860e47828575b5b5
Weipeng Hu
69e1d779860e47828575b5b6
Xudong Jiang
69e1d779860e47828575b5b7
Yap-Peng Tan
69e1d779860e47828575b5b8
Chee Seng Chan
OneHOI: Unifying Human-Object Interaction Generation and Editing
Human-Object Interaction (HOI) modelling captures how humans act upon and relate to objects, typically expressed as <person, action, object> triplets. Existing approaches split into two disjoint families: HOI generation synthesises scenes from structured triplets and layout, but fails to integrate mixed conditions like HOI and object-only entities; and HOI editing modifies interactions via text, yet struggles to decouple pose from physical contact and scale to multiple interactions. We introduce OneHOI, a unified diffusion transformer framework that consolidates HOI generation and editing into a single conditional denoising process driven by shared structured interaction representations. At its core, the Relational Diffusion Transformer (R-DiT) models verb-mediated relations through role- and instance-aware HOI tokens, layout-based spatial Action Grounding, a Structured HOI Attention to enforce interaction topology, and HOI RoPE to disentangle multi-HOI scenes. Trained jointly with modality dropout on our HOI-Edit-44K, along with HOI and object-centric datasets, OneHOI supports layout-guided, layout-free, arbitrary-mask, and mixed-condition control, achieving state-of-the-art results across both HOI generation and editing. Code is available at https://jiuntian.github.io/OneHOI/.
A unified diffusion transformer framework for human-object interaction generation and editing that uses relational modeling and structured attention mechanisms to handle complex interaction scenarios.
diffusion transformer relational diffusion transformer R-DiT HOI tokens layout-based spatial Action Grounding Structured HOI Attention HOI RoPE modality dropout HOI-Edit-44K