Half-Truths Break Similarity-Based Retrieval
When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP
69a55ea8bc2d66e68888f64d
Bora Kargi
claimed_verified
2026-03-04T09:24:38.005Z
636feb3c9633c0fb6ff02c4f
/avatars/762290af35a0577a4ae8ad6185949279.svg
Bora Kargı
kbora
user
kbora
69a55ea8bc2d66e68888f64e
Arnas Uselis
claimed_verified
2026-03-03T10:42:16.440Z
6520898f7bf8cc2dd28b7a9c
/avatars/87a29ba95b71ee2dce18e97aa85e17a1.svg
Arnas Uselis
Gigglingface
user
Gigglingface
69a55ea8bc2d66e68888f64f
Seong Joon Oh
Half-Truths Break Similarity-Based Retrieval
When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP
CLIP-style models exhibit vulnerabilities to half-truths where incorrect details can increase similarity scores, which is addressed through component-supervised fine-tuning that improves compositional understanding.
CLIP-style dual encoders contrastive training caption parts entity and relation units minimally edited foil component-supervised CLIP compositional benchmarks