ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
Retrieval-Augmented Generation (RAG) pipelines must address challenges beyond simple single-document retrieval, such as interpreting visual elements (tables, charts, images), synthesizing information across documents, and providing accurate source grounding. Existing benchmarks fail to capture this complexity, often focusing on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. We introduce ViDoRe v3, a comprehensive multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. It covers 10 datasets across diverse professional domains, comprising ~26,000 document pages paired with 3,099 human-verified queries, each available in 6 languages. Through 12,000 hours of human annotation effort, we provide high-quality annotations for retrieval relevance, bounding box localization, and verified reference answers. Our evaluation of state-of-the-art RAG pipelines reveals that visual retrievers outperform textual ones, late-interaction models and textual reranking substantially improve performance, and hybrid or purely visual contexts enhance answer generation quality. However, current models still struggle with non-textual elements, open-ended queries, and fine-grained visual grounding. To encourage progress in addressing these challenges, the benchmark is released under a commercially permissive license at https://hf.co/vidore.
69676b6fc5e371f6b235d275
António Loison
claimed_verified
2026-01-15T15:03:59.207Z
6377b63b24d97f9f7ec70064
https://cdn-avatars.huggingface.co/v1/production/uploads/6377b63b24d97f9f7ec70064/vBl68PAwwp353MOeOahMH.jpeg
Antonio Loison
antonioloison
user
antonioloison
69676b6fc5e371f6b235d276
Quentin Macé
claimed_verified
2026-01-15T15:04:01.225Z
661e945eebe3616a1b09e279
https://cdn-avatars.huggingface.co/v1/production/uploads/661e945eebe3616a1b09e279/U3DL1BNouUpcusCKAPZm0.jpeg
Quentin Macé
QuentinJG
user
QuentinJG
69676b6fc5e371f6b235d277
Antoine Edy
claimed_verified
2026-01-14T14:21:43.867Z
66e16a677c2eb2da5109fb5c
https://cdn-avatars.huggingface.co/v1/production/uploads/66e16a677c2eb2da5109fb5c/jPkg3g4mtCYbm8ci-ymtX.png
Antoine EDY
antoineedy
user
antoineedy
69676b6fc5e371f6b235d278
Victor Xing
claimed_verified
2026-03-27T11:39:03.405Z
67a8dc9e560939c755f39fb5
https://cdn-avatars.huggingface.co/v1/production/uploads/67a8dc9e560939c755f39fb5/uyfHhPi4fkGPIurIr7SP4.jpeg
Victor Xing
vxing
user
vxing
69676b6fc5e371f6b235d279
Tom Balough
69676b6fc5e371f6b235d27a
Gabriel Moreira
69676b6fc5e371f6b235d27b
Bo Liu
69676b6fc5e371f6b235d27c
Manuel Faysse
69676b6fc5e371f6b235d27d
Céline Hudelot
69676b6fc5e371f6b235d27e
Gautier Viaud
ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
Retrieval-Augmented Generation (RAG) pipelines must address challenges beyond simple single-document retrieval, such as interpreting visual elements (tables, charts, images), synthesizing information across documents, and providing accurate source grounding. Existing benchmarks fail to capture this complexity, often focusing on textual data, single-document comprehension, or evaluating retrieval and generation in isolation. We introduce ViDoRe v3, a comprehensive multimodal RAG benchmark featuring multi-type queries over visually rich document corpora. It covers 10 datasets across diverse professional domains, comprising ~26,000 document pages paired with 3,099 human-verified queries, each available in 6 languages. Through 12,000 hours of human annotation effort, we provide high-quality annotations for retrieval relevance, bounding box localization, and verified reference answers. Our evaluation of state-of-the-art RAG pipelines reveals that visual retrievers outperform textual ones, late-interaction models and textual reranking substantially improve performance, and hybrid or purely visual contexts enhance answer generation quality. However, current models still struggle with non-textual elements, open-ended queries, and fine-grained visual grounding. To encourage progress in addressing these challenges, the benchmark is released under a commercially permissive license at https://hf.co/vidore.
ViDoRe v3 is a multimodal retrieval-augmented generation benchmark with diverse document types and languages, revealing that visual retrieval and late interaction models improve performance while current systems still face challenges with non-textual elements and complex queries.
Retrieval-Augmented Generation multimodal RAG visual retrieval late-interaction models textual reranking hybrid models visual grounding document corpora multi-type queries human annotation