VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to robotic control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are then overlaid directly onto visual observations as structured visual prompts, such as crosshairs and bounding boxes. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%, surpassing competitive baselines including QwenOFT and GR00T-N1.6.
69c3800895b48f16a30078d7
Zixuan Wang
claimed_verified
2026-03-25T10:23:05.364Z
639949a0f7c4648254d1a32b
https://cdn-avatars.huggingface.co/v1/production/uploads/1670990223020-noauth.jpeg
Zixuan WANG
Vincent2311
user
Vincent2311
69c3800895b48f16a30078d8
Yuxin Chen
claimed_verified
2026-03-25T10:23:00.440Z
6795e47005822244e0f8c794
/avatars/7b383f931d7341a3c266ae993306429c.svg
yuxin chen
chen210210203
user
chen210210203
69c3800895b48f16a30078d9
Yuqi Liu
claimed_verified
2026-03-25T10:23:02.353Z
669cefd6119595d21b55a995
/avatars/bafc2387ee70b263bf45c42159381da8.svg
Yuqi Liu
Ricky06662
user
Ricky06662
69c3800895b48f16a30078da
Jinhui Ye
69c3800895b48f16a30078db
Pengguang Chen
69c3800895b48f16a30078dc
Changsheng Lu
69c3800895b48f16a30078dd
Shu Liu
69c3800895b48f16a30078de
Jiaya Jia
VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models
Vision-Language-Action (VLA) models typically map visual observations and linguistic instructions directly to robotic control signals. This "black-box" mapping forces a single forward pass to simultaneously handle instruction interpretation, spatial grounding, and low-level control, often leading to poor spatial precision and limited robustness in out-of-distribution scenarios. To address these limitations, we propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface. Specifically, a "System 2 Planner" decomposes complex instructions into sub-tasks and identifies relevant target objects and goal locations. These spatial anchors are then overlaid directly onto visual observations as structured visual prompts, such as crosshairs and bounding boxes. Guided by these prompts and enhanced by a novel auxiliary visual grounding objective during training, a "System 1 Controller" reliably generates precise low-level execution motions. Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%, surpassing competitive baselines including QwenOFT and GR00T-N1.6.
VP-VLA is a dual-system framework that decouples high-level reasoning from low-level robotic control through structured visual prompting, improving spatial precision and robustness in vision-language-action tasks.
Vision-Language-Action models visual observations linguistic instructions robotic control signals dual-system framework visual prompting interface System 2 Planner System 1 Controller visual grounding objective Robocasa-GR1-Tabletop benchmark SimplerEnv simulation