OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation
In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.
69dda767aea31e227c22c441
Donghao Zhou
claimed_verified
2026-04-15T12:07:58.237Z
67136093d2e50f1e8c9fad52
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0q49MyGuav8lJ9CIeyLhu.png
Donghao Zhou
donghao-zhou
user
donghao-zhou
69dda767aea31e227c22c442
Guisheng Liu
69dda767aea31e227c22c443
Hao Yang
69dda767aea31e227c22c444
Jiatong Li
69dda767aea31e227c22c445
Jingyu Lin
69dda767aea31e227c22c446
Xiaohu Huang
69dda767aea31e227c22c447
Yichen Liu
69dda767aea31e227c22c448
Xin Gao
69dda767aea31e227c22c449
Cunjian Chen
claimed_verified
2026-04-15T12:08:14.609Z
66f2564859a7531df0fd61e3
/avatars/2e08e936b224a0b81862b352334c26f1.svg
Chen
Cunjian
user
Cunjian
69dda767aea31e227c22c44a
Shilei Wen
69dda767aea31e227c22c44b
Chi-Wing Fu
69dda767aea31e227c22c44c
Pheng-Ann Heng
OmniShow: Unifying Multimodal Conditions for Human-Object Interaction Video Generation
In this work, we study Human-Object Interaction Video Generation (HOIVG), which aims to synthesize high-quality human-object interaction videos conditioned on text, reference images, audio, and pose. This task holds significant practical value for automating content creation in real-world applications, such as e-commerce demonstrations, short video production, and interactive entertainment. However, existing approaches fail to accommodate all these requisite conditions. We present OmniShow, an end-to-end framework tailored for this practical yet challenging task, capable of harmonizing multimodal conditions and delivering industry-grade performance. To overcome the trade-off between controllability and quality, we introduce Unified Channel-wise Conditioning for efficient image and pose injection, and Gated Local-Context Attention to ensure precise audio-visual synchronization. To effectively address data scarcity, we develop a Decoupled-Then-Joint Training strategy that leverages a multi-stage training process with model merging to efficiently harness heterogeneous sub-task datasets. Furthermore, to fill the evaluation gap in this field, we establish HOIVG-Bench, a dedicated and comprehensive benchmark for HOIVG. Extensive experiments demonstrate that OmniShow achieves overall state-of-the-art performance across various multimodal conditioning settings, setting a solid standard for the emerging HOIVG task.
OmniShow is an end-to-end framework for human-object interaction video generation that effectively integrates multiple modalities through unified conditioning and attention mechanisms while addressing data scarcity via decoupled training strategies.
Human-Object Interaction Video Generation multimodal conditions end-to-end framework Unified Channel-wise Conditioning Gated Local-Context Attention Decoupled-Then-Joint Training model merging HOIVG-Bench