Lightweight Visual Reasoning for Socially-Aware Robots
Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by 3.3% (less distance), +0.057 description score, and +2.93% accuracy, with less than 3% extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains +0.111,+0.055 and +10.81%,+4.79% on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics
69aa8f38c4f6edddcc6c879f
Alessio Galatolo
claimed_verified
2026-03-06T13:54:43.364Z
64b8e176acfb31c3f7ab2c51
https://cdn-avatars.huggingface.co/v1/production/uploads/64b8e176acfb31c3f7ab2c51/bUwnNQQnfC_1BP4c5080c.jpeg
Alessio Galatolo
alessioGalatolo
user
alessioGalatolo
69aa8f38c4f6edddcc6c87a0
Ronald Cumbal
69aa8f38c4f6edddcc6c87a1
Alexandros Rouchitsas
69aa8f38c4f6edddcc6c87a2
Katie Winkle
69aa8f38c4f6edddcc6c87a3
Didem Gürdür Broo
69aa8f38c4f6edddcc6c87a4
Ginevra Castellano
Lightweight Visual Reasoning for Socially-Aware Robots
Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by 3.3% (less distance), +0.057 description score, and +2.93% accuracy, with less than 3% extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains +0.111,+0.055 and +10.81%,+4.79% on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics
A lightweight language-to-vision feedback module enhances Vision-Language Models for robotics applications by enabling contextual reinterpretation of visual scenes through gated MLP layers.
Vision-Language Models language-to-vision feedback module gated Multi-Layer Perceptron image-token hidden states vision encoder LLM multimodal human-robot interactions Habitat Mementos-Robotics HRI dataset Qwen 2.5 Gemma 3 LLaVA OV 1.5