Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions
We address the problem of tactile localization, where the goal is to identify image regions that share the same material properties as a tactile input. Existing visuo-tactile methods rely on global alignment and thus fail to capture the fine-grained local correspondences required for this task. The challenge is amplified by existing datasets, which predominantly contain close-up, low-diversity images. We propose a model that learns local visuo-tactile alignment via dense cross-modal feature interactions, producing tactile saliency maps for touch-conditioned material segmentation. To overcome dataset constraints, we introduce: (i) in-the-wild multi-material scene images that expand visual diversity, and (ii) a material-diversity pairing strategy that aligns each tactile sample with visually varied yet tactilely consistent images, improving contextual localization and robustness to weak signals. We also construct two new tactile-grounded material segmentation datasets for quantitative evaluation. Experiments on both new and existing benchmarks show that our approach substantially outperforms prior visuo-tactile methods in tactile localization.
69df0111d3c84fcbf2921829
Seongyu Kim
69df0111d3c84fcbf292182a
Seungwoo Lee
69df0111d3c84fcbf292182b
Hyeonggon Ryu
69df0111d3c84fcbf292182c
Joon Son Chung
69df0111d3c84fcbf292182d
Arda Senocak
Seeing Through Touch: Tactile-Driven Visual Localization of Material Regions
We address the problem of tactile localization, where the goal is to identify image regions that share the same material properties as a tactile input. Existing visuo-tactile methods rely on global alignment and thus fail to capture the fine-grained local correspondences required for this task. The challenge is amplified by existing datasets, which predominantly contain close-up, low-diversity images. We propose a model that learns local visuo-tactile alignment via dense cross-modal feature interactions, producing tactile saliency maps for touch-conditioned material segmentation. To overcome dataset constraints, we introduce: (i) in-the-wild multi-material scene images that expand visual diversity, and (ii) a material-diversity pairing strategy that aligns each tactile sample with visually varied yet tactilely consistent images, improving contextual localization and robustness to weak signals. We also construct two new tactile-grounded material segmentation datasets for quantitative evaluation. Experiments on both new and existing benchmarks show that our approach substantially outperforms prior visuo-tactile methods in tactile localization.
Deep learning model for tactile localization that uses dense cross-modal feature interactions to identify material properties in images, overcoming limitations of existing methods through enhanced datasets and material-diversity pairing strategies.
visuo-tactile methods dense cross-modal feature interactions tactile saliency maps material segmentation tactile localization material-diversity pairing strategy