Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive
  Trigger Learning
Multimodal large language models (MLLMs) have advanced embodied agents by
enabling direct perception, reasoning, and planning task-oriented actions from
visual inputs. However, such vision driven embodied agents open a new attack
surface: visual backdoor attacks, where the agent behaves normally until a
visual trigger appears in the scene, then persistently executes an
attacker-specified multi-step policy. We introduce BEAT, the first framework to
inject such visual backdoors into MLLM-based embodied agents using objects in
the environments as triggers. Unlike textual triggers, object triggers exhibit
wide variation across viewpoints and lighting, making them difficult to implant
reliably. BEAT addresses this challenge by (1) constructing a training set that
spans diverse scenes, tasks, and trigger placements to expose agents to trigger
variability, and (2) introducing a two-stage training scheme that first applies
supervised fine-tuning (SFT) and then our novel Contrastive Trigger Learning
(CTL). CTL formulates trigger discrimination as preference learning between
trigger-present and trigger-free inputs, explicitly sharpening the decision
boundaries to ensure precise backdoor activation. Across various embodied agent
benchmarks and MLLMs, BEAT achieves attack success rates up to 80%, while
maintaining strong benign task performance, and generalizes reliably to
out-of-distribution trigger placements. Notably, compared to naive SFT, CTL
boosts backdoor activation accuracy up to 39% under limited backdoor data.
These findings expose a critical yet unexplored security risk in MLLM-based
embodied agents, underscoring the need for robust defenses before real-world
deployment.
6908174c812eca10f9cc5e38
Qiusi Zhan
claimed_verified
2025-11-03T20:52:27.106Z
64f005d8f87b014e81716593
/avatars/cfef589b87bd424e333b2a25063528f8.svg
QIUSI ZHAN
qiusizhan
user
qiusizhan
6908174c812eca10f9cc5e39
Hyeonjeong Ha
6908174c812eca10f9cc5e3a
Rui Yang
claimed_verified
2025-11-03T20:52:31.665Z
64d45451c34a346181b130dd
/avatars/9bb8205b889337df5d321539c9b5d69d.svg
Rui Yang
Ray2333
user
Ray2333
6908174c812eca10f9cc5e3b
Sirui Xu
claimed_verified
2025-11-03T20:52:24.269Z
64f17c31b4344f592fb2821e
/avatars/11c8edb0967491822277a8a0d3ff3d31.svg
Sirui Xu
xusirui
user
xusirui
6908174c812eca10f9cc5e3c
Hanyang Chen
6908174c812eca10f9cc5e3d
Liang-Yan Gui
6908174c812eca10f9cc5e3e
Yu-Xiong Wang
6908174c812eca10f9cc5e3f
Huan Zhang
6908174c812eca10f9cc5e40
Heng Ji
6908174c812eca10f9cc5e41
Daniel Kang
Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive
  Trigger Learning
Multimodal large language models (MLLMs) have advanced embodied agents by
enabling direct perception, reasoning, and planning task-oriented actions from
visual inputs. However, such vision driven embodied agents open a new attack
surface: visual backdoor attacks, where the agent behaves normally until a
visual trigger appears in the scene, then persistently executes an
attacker-specified multi-step policy. We introduce BEAT, the first framework to
inject such visual backdoors into MLLM-based embodied agents using objects in
the environments as triggers. Unlike textual triggers, object triggers exhibit
wide variation across viewpoints and lighting, making them difficult to implant
reliably. BEAT addresses this challenge by (1) constructing a training set that
spans diverse scenes, tasks, and trigger placements to expose agents to trigger
variability, and (2) introducing a two-stage training scheme that first applies
supervised fine-tuning (SFT) and then our novel Contrastive Trigger Learning
(CTL). CTL formulates trigger discrimination as preference learning between
trigger-present and trigger-free inputs, explicitly sharpening the decision
boundaries to ensure precise backdoor activation. Across various embodied agent
benchmarks and MLLMs, BEAT achieves attack success rates up to 80%, while
maintaining strong benign task performance, and generalizes reliably to
out-of-distribution trigger placements. Notably, compared to naive SFT, CTL
boosts backdoor activation accuracy up to 39% under limited backdoor data.
These findings expose a critical yet unexplored security risk in MLLM-based
embodied agents, underscoring the need for robust defenses before real-world
deployment.
BEAT is a framework that injects visual backdoors into MLLM-based embodied agents using object triggers, achieving high attack success rates while maintaining task performance.
multimodal large language models MLLMs embodied agents visual backdoor attacks BEAT Contrastive Trigger Learning CTL supervised fine-tuning SFT preference learning decision boundaries backdoor activation accuracy out-of-distribution trigger placements