Relational Visual Similarity
Humans do not just see attribute similarity -- we also see relational similarity. An apple is like a peach because both are reddish fruit, but the Earth is also like a peach: its crust, mantle, and core correspond to the peach's skin, flesh, and pit. This ability to perceive and recognize relational similarity, is arguable by cognitive scientist to be what distinguishes humans from other species. Yet, all widely used visual similarity metrics today (e.g., LPIPS, CLIP, DINO) focus solely on perceptual attribute similarity and fail to capture the rich, often surprising relational similarities that humans perceive. How can we go beyond the visible content of an image to capture its relational properties? How can we bring images with the same relational logic closer together in representation space? To answer these questions, we first formulate relational image similarity as a measurable problem: two images are relationally similar when their internal relations or functions among visual elements correspond, even if their visual attributes differ. We then curate 114k image-caption dataset in which the captions are anonymized -- describing the underlying relational logic of the scene rather than its surface content. Using this dataset, we finetune a Vision-Language model to measure the relational similarity between images. This model serves as the first step toward connecting images by their underlying relational structure rather than their visible appearance. Our study shows that while relational similarity has a lot of real-world applications, existing image similarity models fail to capture it -- revealing a critical gap in visual computing.
69379fe819d912300c34a313
Thao Nguyen
claimed_verified
2025-12-09T20:17:39.452Z
634ef841de30ee20582b355a
https://cdn-avatars.huggingface.co/v1/production/uploads/634ef841de30ee20582b355a/7W9HHzEjURmUPkQ7U_Nnl.png
Thao Nguyen
thaoshibe
user
thaoshibe
69379fe819d912300c34a314
Sicheng Mo
69379fe819d912300c34a315
Krishna Kumar Singh
69379fe819d912300c34a316
Yilin Wang
claimed_verified
2026-04-01T08:37:18.818Z
63407256733f9eef46941067
/avatars/222bff48087ce6d16de273d69ff56d74.svg
yilin
yilwang
user
yilwang
69379fe819d912300c34a317
Jing Shi
69379fe819d912300c34a318
Nicholas Kolkin
69379fe819d912300c34a319
Eli Shechtman
69379fe819d912300c34a31a
Yong Jae Lee
69379fe819d912300c34a31b
Yuheng Li
Relational Visual Similarity
Humans do not just see attribute similarity -- we also see relational similarity. An apple is like a peach because both are reddish fruit, but the Earth is also like a peach: its crust, mantle, and core correspond to the peach's skin, flesh, and pit. This ability to perceive and recognize relational similarity, is arguable by cognitive scientist to be what distinguishes humans from other species. Yet, all widely used visual similarity metrics today (e.g., LPIPS, CLIP, DINO) focus solely on perceptual attribute similarity and fail to capture the rich, often surprising relational similarities that humans perceive. How can we go beyond the visible content of an image to capture its relational properties? How can we bring images with the same relational logic closer together in representation space? To answer these questions, we first formulate relational image similarity as a measurable problem: two images are relationally similar when their internal relations or functions among visual elements correspond, even if their visual attributes differ. We then curate 114k image-caption dataset in which the captions are anonymized -- describing the underlying relational logic of the scene rather than its surface content. Using this dataset, we finetune a Vision-Language model to measure the relational similarity between images. This model serves as the first step toward connecting images by their underlying relational structure rather than their visible appearance. Our study shows that while relational similarity has a lot of real-world applications, existing image similarity models fail to capture it -- revealing a critical gap in visual computing.
Vision-Language models fine-tuned on anonymized image captions can capture relational similarity between images, a capability lacking in current visual similarity metrics.
relational similarity perceptual attribute similarity LPIPS CLIP DINO Vision-Language model image-caption dataset relational logic representation space