AlignBench: Benchmarking Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs
Assessing image-text alignment models such as CLIP is crucial for bridging visual and linguistic representations. Yet existing benchmarks rely on rule-based perturbations or short captions, limiting their ability to measure fine-grained alignment. We introduce AlignBench, a benchmark that provides a new indicator of image-text alignment by evaluating detailed image-caption pairs generated by diverse image-to-text and text-to-image models. Each sentence is annotated for correctness, enabling direct assessment of VLMs as alignment evaluators. Benchmarking a wide range of decoder-based VLMs reveals three key findings: (i) CLIP-based models, even those tailored for compositional reasoning, remain nearly blind; (ii) detectors systematically over-score early sentences; and (iii) they show strong self-preference, favoring their own outputs and harming detection performance. Our project page will be available at https://dahlian00.github.io/AlignBench/.
692ffb1c26742347f61daf81
Kuniaki Saito
692ffb1c26742347f61daf82
Risa Shinoda
claimed_verified
2025-12-04T08:48:47.311Z
651a7684a1a5e5d617e28f84
/avatars/95f5582df7c13fe4a392006125e6cc0f.svg
Risa Shinoda
risashinoda
user
risashinoda
692ffb1c26742347f61daf83
Shohei Tanaka
692ffb1c26742347f61daf84
Tosho Hirasawa
692ffb1c26742347f61daf85
Fumio Okura
692ffb1c26742347f61daf86
Yoshitaka Ushiku
admin_assigned
2025-12-04T11:12:41.107Z
6268cab5dc80d589a9f8ea82
/avatars/2897993439d641478aa4e797551fccfb.svg
Yoshitaka Ushiku
yushiku
user
yushiku
AlignBench: Benchmarking Fine-Grained Image-Text Alignment with Synthetic Image-Caption Pairs
Assessing image-text alignment models such as CLIP is crucial for bridging visual and linguistic representations. Yet existing benchmarks rely on rule-based perturbations or short captions, limiting their ability to measure fine-grained alignment. We introduce AlignBench, a benchmark that provides a new indicator of image-text alignment by evaluating detailed image-caption pairs generated by diverse image-to-text and text-to-image models. Each sentence is annotated for correctness, enabling direct assessment of VLMs as alignment evaluators. Benchmarking a wide range of decoder-based VLMs reveals three key findings: (i) CLIP-based models, even those tailored for compositional reasoning, remain nearly blind; (ii) detectors systematically over-score early sentences; and (iii) they show strong self-preference, favoring their own outputs and harming detection performance. Our project page will be available at https://dahlian00.github.io/AlignBench/.
AlignBench evaluates image-text models using detailed captions generated by diverse models, revealing insights into their alignment and compositional reasoning capabilities.
CLIP image-text alignment AlignBench image-to-text text-to-image decoder-based VLMs compositional reasoning self-preference