CaptionQA: Is Your Caption as Useful as the Image Itself?
Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains--Natural, Document, E-commerce, and Embodied AI--each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains. The code is available at https://github.com/bronyayang/CaptionQA.
6929091e1f3fae537858b4d9
Shijia Yang
claimed_verified
2025-12-01T09:11:34.204Z
64374254a701a7e744c033a1
/avatars/a103d27fe32c520cb51191920a5b548a.svg
Shijia Yang
shijiay
user
shijiay
6929091e1f3fae537858b4da
Yunong Liu
claimed_verified
2025-12-02T08:49:54.112Z
6276b37886cf863dc36ed3da
/avatars/2cfb6383b6b3e23377df7c82cd2014aa.svg
Yunong Liu
Yunong
user
Yunong
6929091e1f3fae537858b4db
Bohan Zhai
claimed_verified
2025-11-28T13:48:46.450Z
642fb0309b2484d7d857dbae
/avatars/7b10993e665198e80bd6627bf584ed71.svg
Bohan Zhai
Borise
user
Borise
6929091e1f3fae537858b4dc
Ximeng Sun
6929091e1f3fae537858b4dd
Zicheng Liu
6929091e1f3fae537858b4de
Emad Barsoum
6929091e1f3fae537858b4df
Manling Li
claimed_verified
2026-04-13T07:39:46.675Z
6746a140f2ca2162e3bcfe2b
/avatars/d9d8cfb5f112e6ed7f6152fc230135d3.svg
Manling Li
ManlingLi
user
ManlingLi
6929091e1f3fae537858b4e0
Chenfeng Xu
CaptionQA: Is Your Caption as Useful as the Image Itself?
Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, and multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains--Natural, Document, E-commerce, and Embodied AI--each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains. The code is available at https://github.com/bronyayang/CaptionQA.
CaptionQA evaluates caption utility by measuring their effectiveness in supporting downstream tasks, revealing significant gaps compared to traditional image-QA benchmarks.
utility-based benchmark CaptionQA LLM MLLMs image-QA