Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing
Composed Image Retrieval (CIR) is a pivotal and complex task in multimodal understanding. Current CIR benchmarks typically feature limited query categories and fail to capture the diverse requirements of real-world scenarios. To bridge this evaluation gap, we leverage image editing to achieve precise control over modification types and content, enabling a pipeline for synthesizing queries across a broad spectrum of categories. Using this pipeline, we construct EDIR, a novel fine-grained CIR benchmark. EDIR encompasses 5,000 high-quality queries structured across five main categories and fifteen subcategories. Our comprehensive evaluation of 13 multimodal embedding models reveals a significant capability gap; even state-of-the-art models (e.g., RzenEmbed and GME) struggle to perform consistently across all subcategories, highlighting the rigorous nature of our benchmark. Through comparative analysis, we further uncover inherent limitations in existing benchmarks, such as modality biases and insufficient categorical coverage. Furthermore, an in-domain training experiment demonstrates the feasibility of our benchmark. This experiment clarifies the task challenges by distinguishing between categories that are solvable with targeted data and those that expose intrinsic limitations of current model architectures.
6972e18efb12c92b735b73f1
Tingyu Song
claimed_verified
2026-01-23T20:15:16.501Z
64dc29d9b5d625e0e9a6ecb9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/QxGBsnk1cNsBEPqSx4ae-.jpeg
Tingyu Song
songtingyu
user
songtingyu
6972e18efb12c92b735b73f2
Yanzhao Zhang
6972e18efb12c92b735b73f3
Mingxin Li
6972e18efb12c92b735b73f4
Zhuoning Guo
claimed_verified
2026-02-25T17:33:08.598Z
6746b1e2224b22ef67fbff11
https://cdn-avatars.huggingface.co/v1/production/uploads/6746b1e2224b22ef67fbff11/q6sN8lQjtLGCgen_41VxU.jpeg
Zhuoning Guo
Zhuoning
user
Zhuoning
6972e18efb12c92b735b73f5
Dingkun Long
6972e18efb12c92b735b73f6
Pengjun Xie
6972e18efb12c92b735b73f7
Siyue Zhang
claimed_verified
2026-03-06T13:58:06.206Z
638f1803c67af472d317a922
https://cdn-avatars.huggingface.co/v1/production/uploads/638f1803c67af472d317a922/9BMVXqHa-AsdZPmBprcbd.jpeg
siyue zhang
siyue
user
siyue
6972e18efb12c92b735b73f8
Yilun Zhao
claimed_verified
2026-02-06T18:57:58.376Z
62f662bcc58915315c4eccea
https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
Yilun Zhao
yilunzhao
user
yilunzhao
6972e18efb12c92b735b73f9
Shu Wu
Rethinking Composed Image Retrieval Evaluation: A Fine-Grained Benchmark from Image Editing
Composed Image Retrieval (CIR) is a pivotal and complex task in multimodal understanding. Current CIR benchmarks typically feature limited query categories and fail to capture the diverse requirements of real-world scenarios. To bridge this evaluation gap, we leverage image editing to achieve precise control over modification types and content, enabling a pipeline for synthesizing queries across a broad spectrum of categories. Using this pipeline, we construct EDIR, a novel fine-grained CIR benchmark. EDIR encompasses 5,000 high-quality queries structured across five main categories and fifteen subcategories. Our comprehensive evaluation of 13 multimodal embedding models reveals a significant capability gap; even state-of-the-art models (e.g., RzenEmbed and GME) struggle to perform consistently across all subcategories, highlighting the rigorous nature of our benchmark. Through comparative analysis, we further uncover inherent limitations in existing benchmarks, such as modality biases and insufficient categorical coverage. Furthermore, an in-domain training experiment demonstrates the feasibility of our benchmark. This experiment clarifies the task challenges by distinguishing between categories that are solvable with targeted data and those that expose intrinsic limitations of current model architectures.
A novel fine-grained composed image retrieval benchmark is introduced through image editing techniques, revealing significant capability gaps in existing multimodal models and exposing limitations of current benchmarks.
composed image retrieval multimodal embedding models image editing fine-grained benchmark modality biases categorical coverage