AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement
Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability.
692d0d8f4397b1ec214f67e7
Zhizhou Zhong
692d0d8f4397b1ec214f67e8
Yicheng Ji
692d0d8f4397b1ec214f67e9
Zhe Kong
692d0d8f4397b1ec214f67ea
Yiying Liu
admin_assigned
2025-12-01T10:09:18.266Z
667baf38d24677e9afb85871
/avatars/447e05b8a0700128f5f38e34ccfadb39.svg
yiying liu
yiying66
user
yiying66
692d0d8f4397b1ec214f67eb
Jiarui Wang
692d0d8f4397b1ec214f67ec
Jiasun Feng
692d0d8f4397b1ec214f67ed
Lupeng Liu
692d0d8f4397b1ec214f67ee
Xiangyi Wang
692d0d8f4397b1ec214f67ef
Yanjia Li
692d0d8f4397b1ec214f67f0
Yuqing She
692d0d8f4397b1ec214f67f1
Ying Qin
692d0d8f4397b1ec214f67f2
Huan Li
692d0d8f4397b1ec214f67f3
Shuiyang Mao
admin_assigned
2025-12-01T10:10:16.459Z
616446e50a2d89b7d3462615
/avatars/969de84cd239e298bec3c93f7ff8fb0d.svg
shuiyang mao
symao
user
symao
692d0d8f4397b1ec214f67f4
Wei Liu
692d0d8f4397b1ec214f67f5
Wenhan Luo
admin_assigned
2025-12-01T10:10:09.471Z
65f865cf806814aec87d45ce
/avatars/6b931c1efe20422a5985fce6a9e36aed.svg
Wenhan Luo
whluo
user
whluo
AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinement
Recently, multi-person video generation has started to gain prominence. While a few preliminary works have explored audio-driven multi-person talking video generation, they often face challenges due to the high costs of diverse multi-person data collection and the difficulty of driving multiple identities with coherent interactivity. To address these challenges, we propose AnyTalker, a multi-person generation framework that features an extensible multi-stream processing architecture. Specifically, we extend Diffusion Transformer's attention block with a novel identity-aware attention mechanism that iteratively processes identity-audio pairs, allowing arbitrary scaling of drivable identities. Besides, training multi-person generative models demands massive multi-person data. Our proposed training pipeline depends solely on single-person videos to learn multi-person speaking patterns and refines interactivity with only a few real multi-person clips. Furthermore, we contribute a targeted metric and dataset designed to evaluate the naturalness and interactivity of the generated multi-person videos. Extensive experiments demonstrate that AnyTalker achieves remarkable lip synchronization, visual quality, and natural interactivity, striking a favorable balance between data costs and identity scalability.
The proposed AnyTalker framework generates high-quality multi-person talking videos by extending Diffusion Transformer with identity-aware attention, leveraging single-person videos for training, and using a specialized dataset for evaluation.
Diffusion Transformer identity-aware attention multi-person video generation audio-driven generation multi-stream processing lip synchronization visual quality interactivity dataset