Redefining Retrieval Evaluation in the Era of LLMs
Traditional Information Retrieval (IR) metrics, such as nDCG, MAP, and MRR,
assume that human users sequentially examine documents with diminishing
attention to lower ranks. This assumption breaks down in Retrieval Augmented
Generation (RAG) systems, where search results are consumed by Large Language
Models (LLMs), which, unlike humans, process all retrieved documents as a whole
rather than sequentially. Additionally, traditional IR metrics do not account
for related but irrelevant documents that actively degrade generation quality,
rather than merely being ignored. Due to these two major misalignments, namely
human vs. machine position discount and human relevance vs. machine utility,
classical IR metrics do not accurately predict RAG performance. We introduce a
utility-based annotation schema that quantifies both the positive contribution
of relevant passages and the negative impact of distracting ones. Building on
this foundation, we propose UDCG (Utility and Distraction-aware Cumulative
Gain), a metric using an LLM-oriented positional discount to directly optimize
the correlation with the end-to-end answer accuracy. Experiments on five
datasets and six LLMs demonstrate that UDCG improves correlation by up to 36%
compared to traditional metrics. Our work provides a critical step toward
aligning IR evaluation with LLM consumers and enables more reliable assessment
of RAG components
68ff96cf22d452aac6dd41ee
Giovanni Trappolini
68ff96cf22d452aac6dd41ef
Florin Cuconasu
claimed_verified
2025-10-28T15:37:23.492Z
62bfff6788fdef8ecde8c45b
/avatars/637da88b8dce01bdc2a4c1319cc882e3.svg
Florin Cuconasu
florin-hf
user
florin-hf
68ff96cf22d452aac6dd41f0
Simone Filice
68ff96cf22d452aac6dd41f1
Yoelle Maarek
68ff96cf22d452aac6dd41f2
Fabrizio Silvestri
Redefining Retrieval Evaluation in the Era of LLMs
Traditional Information Retrieval (IR) metrics, such as nDCG, MAP, and MRR,
assume that human users sequentially examine documents with diminishing
attention to lower ranks. This assumption breaks down in Retrieval Augmented
Generation (RAG) systems, where search results are consumed by Large Language
Models (LLMs), which, unlike humans, process all retrieved documents as a whole
rather than sequentially. Additionally, traditional IR metrics do not account
for related but irrelevant documents that actively degrade generation quality,
rather than merely being ignored. Due to these two major misalignments, namely
human vs. machine position discount and human relevance vs. machine utility,
classical IR metrics do not accurately predict RAG performance. We introduce a
utility-based annotation schema that quantifies both the positive contribution
of relevant passages and the negative impact of distracting ones. Building on
this foundation, we propose UDCG (Utility and Distraction-aware Cumulative
Gain), a metric using an LLM-oriented positional discount to directly optimize
the correlation with the end-to-end answer accuracy. Experiments on five
datasets and six LLMs demonstrate that UDCG improves correlation by up to 36%
compared to traditional metrics. Our work provides a critical step toward
aligning IR evaluation with LLM consumers and enables more reliable assessment
of RAG components
A new metric, UDCG, is introduced to better evaluate Retrieval Augmented Generation systems by accounting for both the utility of relevant documents and the distraction of irrelevant ones, improving correlation with end-to-end answer accuracy.
Retrieval Augmented Generation Large Language Models nDCG MAP MRR utility-based annotation schema UDCG positional discount end-to-end answer accuracy