RoboOmni: Proactive Robot Manipulation in Omni-modal Context
Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid
progress in Vision-Language-Action (VLA) models for robotic manipulation.
Although effective in many scenarios, current approaches largely rely on
explicit instructions, whereas in real-world interactions, humans rarely issue
instructions directly. Effective collaboration requires robots to infer user
intentions proactively. In this work, we introduce cross-modal contextual
instructions, a new setting where intent is derived from spoken dialogue,
environmental sounds, and visual cues rather than explicit commands. To address
this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor
framework based on end-to-end omni-modal LLMs that unifies intention
recognition, interaction confirmation, and action execution. RoboOmni fuses
auditory and visual signals spatiotemporally for robust intention recognition,
while supporting direct speech interaction. To address the absence of training
data for proactive intention recognition in robotic manipulation, we build
OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640
backgrounds, and six contextual instruction types. Experiments in simulation
and real-world settings show that RoboOmni surpasses text- and ASR-based
baselines in success rate, inference speed, intention recognition, and
proactive assistance.
690168ee646208eac0d1f2fb
Siyin Wang
claimed_verified
2025-10-29T12:46:55.872Z
64c3c631e77ea9f28111172a
/avatars/495dbb73b69c399bae780da3118e332f.svg
Siyin Wang (SII)
sinwang
user
sinwang
690168ee646208eac0d1f2fc
Jinlan Fu
claimed_verified
2026-01-20T14:48:50.265Z
618497ea8aaadc9253c2dfa9
/avatars/2eb3954a99f5aede6f31b8ae49b8c910.svg
Fu Jinlan
Jinlan
user
Jinlan
690168ee646208eac0d1f2fd
Feihong Liu
690168ee646208eac0d1f2fe
Xinzhe He
690168ee646208eac0d1f2ff
Huangxuan Wu
690168ee646208eac0d1f300
Junhao Shi
claimed_verified
2025-12-25T20:54:32.682Z
64ff0de941444912cdd97a21
https://cdn-avatars.huggingface.co/v1/production/uploads/64ff0de941444912cdd97a21/Q5ZYJo7SWoIDEkuUl2Ul0.jpeg
Michael Shi (SII)
Michael0354
user
Michael0354
690168ee646208eac0d1f301
Kexin Huang
690168ee646208eac0d1f302
Zhaoye Fei
claimed_verified
2026-02-13T09:39:20.615Z
629ef8544313a7c1dd671130
https://cdn-avatars.huggingface.co/v1/production/uploads/629ef8544313a7c1dd671130/i5xfHIgELcuO1Ew19ebTw.png
Zhaoye Fei
ngc7293
user
ngc7293
690168ee646208eac0d1f303
Jingjing Gong
690168ee646208eac0d1f304
Zuxuan Wu
690168ee646208eac0d1f305
Yugang Jiang
690168ee646208eac0d1f306
See-Kiong Ng
690168ee646208eac0d1f307
Tat-Seng Chua
690168ee646208eac0d1f308
Xipeng Qiu
RoboOmni: Proactive Robot Manipulation in Omni-modal Context
Recent advances in Multimodal Large Language Models (MLLMs) have driven rapid
progress in Vision-Language-Action (VLA) models for robotic manipulation.
Although effective in many scenarios, current approaches largely rely on
explicit instructions, whereas in real-world interactions, humans rarely issue
instructions directly. Effective collaboration requires robots to infer user
intentions proactively. In this work, we introduce cross-modal contextual
instructions, a new setting where intent is derived from spoken dialogue,
environmental sounds, and visual cues rather than explicit commands. To address
this new setting, we present RoboOmni, a Perceiver-Thinker-Talker-Executor
framework based on end-to-end omni-modal LLMs that unifies intention
recognition, interaction confirmation, and action execution. RoboOmni fuses
auditory and visual signals spatiotemporally for robust intention recognition,
while supporting direct speech interaction. To address the absence of training
data for proactive intention recognition in robotic manipulation, we build
OmniAction, comprising 140k episodes, 5k+ speakers, 2.4k event sounds, 640
backgrounds, and six contextual instruction types. Experiments in simulation
and real-world settings show that RoboOmni surpasses text- and ASR-based
baselines in success rate, inference speed, intention recognition, and
proactive assistance.
RoboOmni, a Perceiver-Thinker-Talker-Executor framework using end-to-end omni-modal LLMs, improves robotic manipulation by inferring user intentions from spoken dialogue, environmental sounds, and visual cues.
Multimodal Large Language Models Vision-Language-Action models cross-modal contextual instructions Perceiver-Thinker-Talker-Executor end-to-end omni-modal LLMs intention recognition interaction confirmation action execution spatiotemporal fusion OmniAction proactive intention recognition text-based baselines ASR-based baselines