Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatars
Generating talking avatars is a fundamental task in video generation. Although existing methods can generate full-body talking avatars with simple human motion, extending this task to grounded human-object interaction (GHOI) remains an open challenge, requiring the avatar to perform text-aligned interactions with surrounding objects. This challenge stems from the need for environmental perception and the control-quality dilemma in GHOI generation. To address this, we propose a novel dual-stream framework, InteractAvatar, which decouples perception and planning from video synthesis for grounded human-object interaction. Leveraging detection to enhance environmental perception, we introduce a Perception and Interaction Module (PIM) to generate text-aligned interaction motions. Additionally, an Audio-Interaction Aware Generation Module (AIM) is proposed to synthesize vivid talking avatars performing object interactions. With a specially designed motion-to-video aligner, PIM and AIM share a similar network structure and enable parallel co-generation of motions and plausible videos, effectively mitigating the control-quality dilemma. Finally, we establish a benchmark, GroundedInter, for evaluating GHOI video generation. Extensive experiments and comparisons demonstrate the effectiveness of our method in generating grounded human-object interactions for talking avatars. Project page: https://interactavatar.github.io
6981707cce18b18628096000
Youliang Zhang
claimed_verified
2026-02-03T10:07:40.952Z
67d3a681536175c33a85c7ed
/avatars/7fd5969c7eed2127f45bf5a9f4189280.svg
zliang
youliang1233214
user
youliang1233214
6981707cce18b18628096001
Zhengguang Zhou
6981707cce18b18628096002
Zhentao Yu
6981707cce18b18628096003
Ziyao Huang
6981707cce18b18628096004
Teng Hu
6981707cce18b18628096005
Sen Liang
6981707cce18b18628096006
Guozhen Zhang
6981707cce18b18628096007
Ziqiao Peng
6981707cce18b18628096008
Shunkai Li
6981707cce18b18628096009
Yi Chen
6981707cce18b1862809600a
Zixiang Zhou
6981707cce18b1862809600b
Yuan Zhou
6981707cce18b1862809600c
Qinglin Lu
6981707cce18b1862809600d
Xiu Li
Making Avatars Interact: Towards Text-Driven Human-Object Interaction for Controllable Talking Avatars
Generating talking avatars is a fundamental task in video generation. Although existing methods can generate full-body talking avatars with simple human motion, extending this task to grounded human-object interaction (GHOI) remains an open challenge, requiring the avatar to perform text-aligned interactions with surrounding objects. This challenge stems from the need for environmental perception and the control-quality dilemma in GHOI generation. To address this, we propose a novel dual-stream framework, InteractAvatar, which decouples perception and planning from video synthesis for grounded human-object interaction. Leveraging detection to enhance environmental perception, we introduce a Perception and Interaction Module (PIM) to generate text-aligned interaction motions. Additionally, an Audio-Interaction Aware Generation Module (AIM) is proposed to synthesize vivid talking avatars performing object interactions. With a specially designed motion-to-video aligner, PIM and AIM share a similar network structure and enable parallel co-generation of motions and plausible videos, effectively mitigating the control-quality dilemma. Finally, we establish a benchmark, GroundedInter, for evaluating GHOI video generation. Extensive experiments and comparisons demonstrate the effectiveness of our method in generating grounded human-object interactions for talking avatars. Project page: https://interactavatar.github.io
A dual-stream framework called InteractAvatar is presented for generating talking avatars that can interact with objects in their environment, addressing challenges in grounded human-object interaction through decoupled perception and planning modules.
dual-stream framework perception and interaction module audio-interaction aware generation module motion-to-video aligner grounded human-object interaction talking avatars