VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing,
  Speaking, and Acting
Current Vision-Language-Action (VLA) models are often constrained by a rigid,
static interaction paradigm, which lacks the ability to see, hear, speak, and
act concurrently as well as handle real-time user interruptions dynamically.
This hinders seamless embodied collaboration, resulting in an inflexible and
unresponsive user experience. To address these limitations, we introduce
VITA-E, a novel embodied interaction framework designed for both behavioral
concurrency and nearly real-time interruption. The core of our approach is a
dual-model architecture where two parallel VLA instances operate as an ``Active
Model'' and a ``Standby Model'', allowing the embodied agent to observe its
environment, listen to user speech, provide verbal responses, and execute
actions, all concurrently and interruptibly, mimicking human-like multitasking
capabilities. We further propose a ``model-as-controller'' paradigm, where we
fine-tune the VLM to generate special tokens that serve as direct system-level
commands, coupling the model's reasoning with the system's behavior.
Experiments conducted on a physical humanoid platform demonstrate that VITA-E
can reliably handle complex interactive scenarios. Our framework is compatible
with various dual-system VLA models, achieving an extremely high success rate
on emergency stops and speech interruptions while also successfully performing
concurrent speech and action. This represents a significant step towards more
natural and capable embodied assistants.
69002d2622d452aac6dd42d4
Xiaoyu Liu
69002d2622d452aac6dd42d5
Chaoyou Fu
69002d2622d452aac6dd42d6
Chi Yan
69002d2622d452aac6dd42d7
Chu Wu
69002d2622d452aac6dd42d8
Haihan Gao
69002d2622d452aac6dd42d9
Yi-Fan Zhang
claimed_verified
2025-12-10T09:10:53.167Z
623d8ca4c29adf5ef6175615
https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
Yi-Fan Zhang
yifanzhang114
user
yifanzhang114
69002d2622d452aac6dd42da
Shaoqi Dong
69002d2622d452aac6dd42db
Cheng Qian
69002d2622d452aac6dd42dc
Bin Luo
69002d2622d452aac6dd42dd
Xiuyong Yang
69002d2622d452aac6dd42de
Guanwu Li
69002d2622d452aac6dd42df
Yusheng Cai
69002d2622d452aac6dd42e0
Yunhang Shen
claimed_verified
2026-04-08T08:08:29.076Z
6483143902f98c3f05aff915
https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
沈云航 Yunhang Shen
shenyunhang
user
shenyunhang
69002d2622d452aac6dd42e1
Deqiang Jiang
69002d2622d452aac6dd42e2
Haoyu Cao
claimed_verified
2026-01-30T09:38:02.790Z
641c1a85999935676ec7ead4
/avatars/738a4e43bb3ddd4de287acfa26095b3d.svg
caohaoyu
rechy
user
rechy
69002d2622d452aac6dd42e3
Xing Sun
claimed_verified
2025-10-28T15:36:42.216Z
647401e50da364bd0d002f2a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/vPuPn7EV092mLBOM2YZXd.png
XING SUN
tedsun
user
tedsun
69002d2622d452aac6dd42e4
Caifeng Shan
69002d2622d452aac6dd42e5
Ran He
VITA-E: Natural Embodied Interaction with Concurrent Seeing, Hearing,
  Speaking, and Acting
Current Vision-Language-Action (VLA) models are often constrained by a rigid,
static interaction paradigm, which lacks the ability to see, hear, speak, and
act concurrently as well as handle real-time user interruptions dynamically.
This hinders seamless embodied collaboration, resulting in an inflexible and
unresponsive user experience. To address these limitations, we introduce
VITA-E, a novel embodied interaction framework designed for both behavioral
concurrency and nearly real-time interruption. The core of our approach is a
dual-model architecture where two parallel VLA instances operate as an ``Active
Model'' and a ``Standby Model'', allowing the embodied agent to observe its
environment, listen to user speech, provide verbal responses, and execute
actions, all concurrently and interruptibly, mimicking human-like multitasking
capabilities. We further propose a ``model-as-controller'' paradigm, where we
fine-tune the VLM to generate special tokens that serve as direct system-level
commands, coupling the model's reasoning with the system's behavior.
Experiments conducted on a physical humanoid platform demonstrate that VITA-E
can reliably handle complex interactive scenarios. Our framework is compatible
with various dual-system VLA models, achieving an extremely high success rate
on emergency stops and speech interruptions while also successfully performing
concurrent speech and action. This represents a significant step towards more
natural and capable embodied assistants.
VITA-E, a dual-model embodied interaction framework, enables concurrent and interruptible vision-language-action capabilities, enhancing real-time user interaction.
embodied interaction framework dual-model architecture Active Model Standby Model model-as-controller VLM special tokens system-level commands emergency stops speech interruptions concurrent speech and action