Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models
Omni-modal large language models (OLLMs) aim to unify multimodal understanding and generation, yet incorporating speech with 3D facial animation remains largely unexplored despite its importance for natural interaction. A key challenge arises from the representation mismatch between discrete, token-level semantic reasoning in LLMs and the dense, fine-grained temporal dynamics required for 3D facial motion, which makes direct modeling difficult to optimize under limited data. We propose Expressive Omni (Ex-Omni), an open-source omni-modal framework that augments OLLMs with speech-accompanied 3D facial animation. Ex-Omni reduces learning difficulty by decoupling semantic reasoning from temporal generation, leveraging speech units as temporal scaffolding and a unified token-as-query gated fusion (TQGF) mechanism for controlled semantic injection. We further introduce InstructEx, a dataset aims to facilitate augment OLLMs with speech-accompanied 3D facial animation. Extensive experiments demonstrate that Ex-Omni performs competitively against existing open-source OLLMs while enabling stable aligned speech and facial animation generation.
698d49ec65c0d15a6d162170
Haoyu Zhang
claimed_verified
2026-02-12T13:28:21.307Z
665d72007bef1cfc313a92dd
/avatars/6d56671153bbf1ffff072472678819da.svg
Haoyu Zhang
lemonade666
user
lemonade666
698d49ec65c0d15a6d162171
Zhipeng Li
698d49ec65c0d15a6d162172
Yiwen Guo
698d49ec65c0d15a6d162173
Tianshu Yu
Ex-Omni: Enabling 3D Facial Animation Generation for Omni-modal Large Language Models
Omni-modal large language models (OLLMs) aim to unify multimodal understanding and generation, yet incorporating speech with 3D facial animation remains largely unexplored despite its importance for natural interaction. A key challenge arises from the representation mismatch between discrete, token-level semantic reasoning in LLMs and the dense, fine-grained temporal dynamics required for 3D facial motion, which makes direct modeling difficult to optimize under limited data. We propose Expressive Omni (Ex-Omni), an open-source omni-modal framework that augments OLLMs with speech-accompanied 3D facial animation. Ex-Omni reduces learning difficulty by decoupling semantic reasoning from temporal generation, leveraging speech units as temporal scaffolding and a unified token-as-query gated fusion (TQGF) mechanism for controlled semantic injection. We further introduce InstructEx, a dataset aims to facilitate augment OLLMs with speech-accompanied 3D facial animation. Extensive experiments demonstrate that Ex-Omni performs competitively against existing open-source OLLMs while enabling stable aligned speech and facial animation generation.
Ex-Omni is an open-source framework that enhances omni-modal large language models with speech-accompanied 3D facial animation by decoupling semantic reasoning from temporal generation and using speech units as temporal scaffolding.
omni-modal large language models 3D facial animation speech units token-as-query gated fusion temporal scaffolding semantic reasoning temporal generation