Embodied Referring Expression Comprehension in Human-Robot Interaction
As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.
6937981c19d912300c34a29c
Md Mofijul Islam
admin_assigned
2025-12-09T11:50:20.500Z
60d599155a14347cd1f6c6eb
/avatars/f469012fd13377fb989810d56551e395.svg
Md Mofijul Islam
mmiakashs
user
mmiakashs
6937981c19d912300c34a29d
Alexi Gladstone
admin_assigned
2025-12-09T11:50:28.336Z
63fbbff41b4b1bd4e7042fd1
/avatars/d186d0859046b35723fe1156d8f68917.svg
Alexi Gladstone
alexiglad
user
alexiglad
6937981c19d912300c34a29e
Sujan Sarker
admin_assigned
2025-12-09T11:50:34.271Z
65ca56e011a80579a4a63db3
/avatars/a35ffb0486f79e1cc8575bdda58de22c.svg
sujan sarker
Anubis91
user
Anubis91
6937981c19d912300c34a29f
Ganesh Nanduru
6937981c19d912300c34a2a0
Md Fahim
6937981c19d912300c34a2a1
Keyan Du
6937981c19d912300c34a2a2
Aman Chadha
claimed_verified
2025-12-09T09:37:59.152Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
amanchadha
user
amanchadha
6937981c19d912300c34a2a3
Tariq Iqbal
Embodied Referring Expression Comprehension in Human-Robot Interaction
As robots enter human workspaces, there is a crucial need for them to comprehend embodied human instructions, enabling intuitive and fluent human-robot interaction (HRI). However, accurate comprehension is challenging due to a lack of large-scale datasets that capture natural embodied interactions in diverse HRI settings. Existing datasets suffer from perspective bias, single-view collection, inadequate coverage of nonverbal gestures, and a predominant focus on indoor environments. To address these issues, we present the Refer360 dataset, a large-scale dataset of embodied verbal and nonverbal interactions collected across diverse viewpoints in both indoor and outdoor settings. Additionally, we introduce MuRes, a multimodal guided residual module designed to improve embodied referring expression comprehension. MuRes acts as an information bottleneck, extracting salient modality-specific signals and reinforcing them into pre-trained representations to form complementary features for downstream tasks. We conduct extensive experiments on four HRI datasets, including the Refer360 dataset, and demonstrate that current multimodal models fail to capture embodied interactions comprehensively; however, augmenting them with MuRes consistently improves performance. These findings establish Refer360 as a valuable benchmark and exhibit the potential of guided residual learning to advance embodied referring expression comprehension in robots operating within human environments.
A large-scale dataset and multimodal model improve embodied interaction comprehension in robots by addressing perspective bias and enhancing multimodal signal integration.
Refer360 MuRes multimodal guided residual module embodied referring expression comprehension information bottleneck salient modality-specific signals pre-trained representations HRI datasets