Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics
Human conversation involves continuous exchanges of speech and nonverbal cues such as head nods, gaze shifts, and facial expressions that convey attention and emotion. Modeling these bidirectional dynamics in 3D is essential for building expressive avatars and interactive robots. However, existing frameworks often treat talking and listening as independent processes or rely on non-causal full-sequence modeling, hindering temporal coherence across turns. We present TIMAR (Turn-level Interleaved Masked AutoRegression), a causal framework for 3D conversational head generation that models dialogue as interleaved audio-visual contexts. It fuses multimodal information within each turn and applies turn-level causal attention to accumulate conversational history, while a lightweight diffusion head predicts continuous 3D head dynamics that captures both coordination and expressive variability. Experiments on the DualTalk benchmark show that TIMAR reduces Fréchet Distance and MSE by 15-30% on the test set, and achieves similar gains on out-of-distribution data. The source code will be released in the GitHub repository https://github.com/CoderChen01/towards-seamleass-interaction.
694360bd542d62d58a7bf61f
Junjie Chen
claimed_verified
2025-12-19T09:07:23.792Z
652f8642338c761caf474169
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/mq5jjqqNaFxVboWGDEocJ.jpeg
Junjie Chen
coderchen01
user
coderchen01
694360bd542d62d58a7bf620
Fei Wang
694360bd542d62d58a7bf621
Zhihao Huang
694360bd542d62d58a7bf622
Qing Zhou
694360bd542d62d58a7bf623
Kun Li
694360bd542d62d58a7bf624
Dan Guo
694360bd542d62d58a7bf625
Linfeng Zhang
694360bd542d62d58a7bf626
Xun Yang
Towards Seamless Interaction: Causal Turn-Level Modeling of Interactive 3D Conversational Head Dynamics
Human conversation involves continuous exchanges of speech and nonverbal cues such as head nods, gaze shifts, and facial expressions that convey attention and emotion. Modeling these bidirectional dynamics in 3D is essential for building expressive avatars and interactive robots. However, existing frameworks often treat talking and listening as independent processes or rely on non-causal full-sequence modeling, hindering temporal coherence across turns. We present TIMAR (Turn-level Interleaved Masked AutoRegression), a causal framework for 3D conversational head generation that models dialogue as interleaved audio-visual contexts. It fuses multimodal information within each turn and applies turn-level causal attention to accumulate conversational history, while a lightweight diffusion head predicts continuous 3D head dynamics that captures both coordination and expressive variability. Experiments on the DualTalk benchmark show that TIMAR reduces Fréchet Distance and MSE by 15-30% on the test set, and achieves similar gains on out-of-distribution data. The source code will be released in the GitHub repository https://github.com/CoderChen01/towards-seamleass-interaction.
TIMAR, a causal framework for 3D conversational head generation, models dialogue as interleaved audio-visual contexts and predicts continuous 3D head dynamics, improving coherence and expressive variability.
Turn-level Interleaved Masked AutoRegression causal framework 3D conversational head generation dialogue interleaved audio-visual contexts turn-level causal attention conversational history lightweight diffusion head Fréchet Distance MSE DualTalk benchmark