CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation
As language models shift from single-shot answer generation toward multi-step reasoning that retrieves and consumes evidence mid-inference, evaluating the role of individual retrieved items becomes more important. Existing RAG evaluation typically targets final-answer quality, citation faithfulness, or answer-level attribution, but none of these directly targets the intervention-based, per-evidence-item utility view we study here. We introduce CUE-R, a lightweight intervention-based framework for measuring per-evidence-item operational utility in single-shot RAG using shallow observable retrieval-use traces. CUE-R perturbs individual evidence items via REMOVE, REPLACE, and DUPLICATE operators, then measures changes along three utility axes (correctness, proxy-based grounding faithfulness, and confidence error) plus a trace-divergence signal. We also outline an operational evidence-role taxonomy for interpreting intervention outcomes. Experiments on HotpotQA and 2WikiMultihopQA with Qwen-3 8B and GPT-5.2 reveal a consistent pattern: REMOVE and REPLACE substantially harm correctness and grounding while producing large trace shifts, whereas DUPLICATE is often answer-redundant yet not fully behaviorally neutral. A zero-retrieval control confirms that these effects arise from degradation of meaningful retrieval. A two-support ablation further shows that multi-hop evidence items can interact non-additively: removing both supports harms performance far more than either single removal. Our results suggest that answer-only evaluation misses important evidence effects and that intervention-based utility analysis is a practical complement for RAG evaluation.
69d5af10b9b236b710ae8073
Siddharth Jain
claimed_verified
2026-04-08T08:05:31.826Z
6418a84922270b3ccf145d6d
/avatars/9fbf669c39b9dd6a48700e0636c08587.svg
Sid
jainsid24
user
jainsid24
69d5af10b9b236b710ae8074
Venkat Narayan Vedam
claimed_verified
2026-04-09T08:07:21.859Z
69d71c972174044cf08d8a02
/avatars/055f87b9a2f135b0f3109f98b4bcd67f.svg
Venkat Vedam
vnvenkat
user
vnvenkat
CUE-R: Beyond the Final Answer in Retrieval-Augmented Generation
As language models shift from single-shot answer generation toward multi-step reasoning that retrieves and consumes evidence mid-inference, evaluating the role of individual retrieved items becomes more important. Existing RAG evaluation typically targets final-answer quality, citation faithfulness, or answer-level attribution, but none of these directly targets the intervention-based, per-evidence-item utility view we study here. We introduce CUE-R, a lightweight intervention-based framework for measuring per-evidence-item operational utility in single-shot RAG using shallow observable retrieval-use traces. CUE-R perturbs individual evidence items via REMOVE, REPLACE, and DUPLICATE operators, then measures changes along three utility axes (correctness, proxy-based grounding faithfulness, and confidence error) plus a trace-divergence signal. We also outline an operational evidence-role taxonomy for interpreting intervention outcomes. Experiments on HotpotQA and 2WikiMultihopQA with Qwen-3 8B and GPT-5.2 reveal a consistent pattern: REMOVE and REPLACE substantially harm correctness and grounding while producing large trace shifts, whereas DUPLICATE is often answer-redundant yet not fully behaviorally neutral. A zero-retrieval control confirms that these effects arise from degradation of meaningful retrieval. A two-support ablation further shows that multi-hop evidence items can interact non-additively: removing both supports harms performance far more than either single removal. Our results suggest that answer-only evaluation misses important evidence effects and that intervention-based utility analysis is a practical complement for RAG evaluation.
Researchers developed a framework to measure the operational utility of individual retrieved items in retrieval-augmented generation systems by perturbing evidence and analyzing changes in correctness, grounding faithfulness, and confidence error.
retrieval-augmented generation evidence items intervention-based framework operational utility shallow observable retrieval-use traces REMOVE operator REPLACE operator DUPLICATE operator correctness grounding faithfulness confidence error trace-divergence signal evidence-role taxonomy HotpotQA 2WikiMultihopQA Qwen-3 8B GPT-5.2 multi-hop evidence items answer-only evaluation