CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmark
Wearable devices such as smart glasses are transforming the way people
interact with their surroundings, enabling users to seek information regarding
entities in their view. Multi-Modal Retrieval-Augmented Generation (MM-RAG)
plays a key role in supporting such questions, yet there is still no
comprehensive benchmark for this task, especially regarding wearables
scenarios. To fill this gap, we present CRAG-MM -- a Comprehensive RAG
benchmark for Multi-modal Multi-turn conversations. CRAG-MM contains a diverse
set of 6.5K (image, question, answer) triplets and 2K visual-based multi-turn
conversations across 13 domains, including 6.2K egocentric images designed to
mimic captures from wearable devices. We carefully constructed the questions to
reflect real-world scenarios and challenges, including five types of
image-quality issues, six question types, varying entity popularity, differing
information dynamism, and different conversation turns. We design three tasks:
single-source augmentation, multi-source augmentation, and multi-turn
conversations -- each paired with an associated retrieval corpus and APIs for
both image-KG retrieval and webpage retrieval. Our evaluation shows that
straightforward RAG approaches achieve only 32% and 43% truthfulness on CRAG-MM
single- and multi-turn QA, respectively, whereas state-of-the-art industry
solutions have similar quality (32%/45%), underscoring ample room for
improvement. The benchmark has hosted KDD Cup 2025, attracting about 1K
participants and 5K submissions, with winning solutions improving baseline
performance by 28%, highlighting its early impact on advancing the field.
690419172c556835fa67f038
Jiaqi Wang
690419172c556835fa67f039
Xiao Yang
690419172c556835fa67f03a
Kai Sun
690419172c556835fa67f03b
Parth Suresh
690419172c556835fa67f03c
Sanat Sharma
claimed_verified
2025-11-10T09:32:48.742Z
64beaa2fae436c88133e240d
/avatars/db774990265b98d8f6c6909819c92ddb.svg
Sanat Sharma
sanatsharma
user
sanatsharma
690419172c556835fa67f03d
Adam Czyzewski
690419172c556835fa67f03e
Derek Andersen
690419172c556835fa67f03f
Surya Appini
690419172c556835fa67f040
Arkav Banerjee
690419172c556835fa67f041
Sajal Choudhary
690419172c556835fa67f042
Shervin Ghasemlou
690419172c556835fa67f043
Ziqiang Guan
690419172c556835fa67f044
Akil Iyer
690419172c556835fa67f045
Haidar Khan
690419172c556835fa67f046
Lingkun Kong
690419172c556835fa67f047
Roy Luo
690419172c556835fa67f048
Tiffany Ma
690419172c556835fa67f049
Zhen Qiao
690419172c556835fa67f04a
David Tran
690419172c556835fa67f04b
Wenfang Xu
690419172c556835fa67f04c
Skyler Yeatman
690419172c556835fa67f04d
Chen Zhou
690419172c556835fa67f04e
Gunveer Gujral
690419172c556835fa67f04f
Yinglong Xia
690419172c556835fa67f050
Shane Moon
690419172c556835fa67f051
Nicolas Scheffer
690419172c556835fa67f052
Nirav Shah
690419172c556835fa67f053
Eun Chang
690419172c556835fa67f054
Yue Liu
690419172c556835fa67f055
Florian Metze
690419172c556835fa67f056
Tammy Stark
690419172c556835fa67f057
Zhaleh Feizollahi
690419172c556835fa67f058
Andrea Jessee
690419172c556835fa67f059
Mangesh Pujari
690419172c556835fa67f05a
Ahmed Aly
690419172c556835fa67f05b
Babak Damavandi
690419172c556835fa67f05c
Rakesh Wanga
690419172c556835fa67f05d
Anuj Kumar
690419172c556835fa67f05e
Rohit Patel
690419172c556835fa67f05f
Wen-tau Yih
690419172c556835fa67f060
Xin Luna Dong
CRAG-MM: Multi-modal Multi-turn Comprehensive RAG Benchmark
Wearable devices such as smart glasses are transforming the way people
interact with their surroundings, enabling users to seek information regarding
entities in their view. Multi-Modal Retrieval-Augmented Generation (MM-RAG)
plays a key role in supporting such questions, yet there is still no
comprehensive benchmark for this task, especially regarding wearables
scenarios. To fill this gap, we present CRAG-MM -- a Comprehensive RAG
benchmark for Multi-modal Multi-turn conversations. CRAG-MM contains a diverse
set of 6.5K (image, question, answer) triplets and 2K visual-based multi-turn
conversations across 13 domains, including 6.2K egocentric images designed to
mimic captures from wearable devices. We carefully constructed the questions to
reflect real-world scenarios and challenges, including five types of
image-quality issues, six question types, varying entity popularity, differing
information dynamism, and different conversation turns. We design three tasks:
single-source augmentation, multi-source augmentation, and multi-turn
conversations -- each paired with an associated retrieval corpus and APIs for
both image-KG retrieval and webpage retrieval. Our evaluation shows that
straightforward RAG approaches achieve only 32% and 43% truthfulness on CRAG-MM
single- and multi-turn QA, respectively, whereas state-of-the-art industry
solutions have similar quality (32%/45%), underscoring ample room for
improvement. The benchmark has hosted KDD Cup 2025, attracting about 1K
participants and 5K submissions, with winning solutions improving baseline
performance by 28%, highlighting its early impact on advancing the field.
A comprehensive benchmark, CRAG-MM, for multi-modal multi-turn conversations in wearable device scenarios is introduced, highlighting the need for improved retrieval-augmented generation approaches.
Multi-Modal Retrieval-Augmented Generation MM-RAG Comprehensive RAG benchmark multi-modal multi-turn conversations egocentric images image-KG retrieval webpage retrieval single-source augmentation multi-source augmentation