DR^{3}-Eval: Towards Realistic and Reproducible Deep Research Evaluation
Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR^{3}-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR^{3}-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR^{3}-Agent based on multiple state-of-the-art language models demonstrate that DR^{3}-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.
69e196b8860e47828575b408
Qianqian Xie
69e196b8860e47828575b409
Qingheng Xiong
69e196b8860e47828575b40a
He Zhu
69e196b8860e47828575b40b
Tiantian Xia
69e196b8860e47828575b40c
Xueming Han
69e196b8860e47828575b40d
Fanyu Meng
69e196b8860e47828575b40e
Jiakai Wang
69e196b8860e47828575b40f
Zhiqi Bai
69e196b8860e47828575b410
Chengkang Jiang
69e196b8860e47828575b411
Zhaohui Wang
69e196b8860e47828575b412
Yubin Guo
69e196b8860e47828575b413
Yuqing Wen
69e196b8860e47828575b414
Jiayang Mao
claimed_verified
2026-04-17T06:56:54.746Z
66016639b1e509e1e4cb37b0
/avatars/805fcd504a0d6a740783792263f28c9a.svg
Jiayang Mao
Nucleon-17th
user
Nucleon-17th
69e196b8860e47828575b415
Zijie Zhang
69e196b8860e47828575b416
Shihao Li
69e196b8860e47828575b417
Yanghai Wang
69e196b8860e47828575b418
Yuxiang Ren
69e196b8860e47828575b419
Junlan Feng
69e196b8860e47828575b41a
Jiaheng Liu
DR^{3}-Eval: Towards Realistic and Reproducible Deep Research Evaluation
Deep Research Agents (DRAs) aim to solve complex, long-horizon research tasks involving planning, retrieval, multimodal understanding, and report generation, yet their evaluation remains challenging due to dynamic web environments and ambiguous task definitions. We propose DR^{3}-Eval, a realistic and reproducible benchmark for evaluating deep research agents on multimodal, multi-file report generation. DR^{3}-Eval is constructed from authentic user-provided materials and paired with a per-task static research sandbox corpus that simulates open-web complexity while remaining fully verifiable, containing supportive documents, distractors, and noise. Moreover, we introduce a multi-dimensional evaluation framework measuring Information Recall, Factual Accuracy, Citation Coverage, Instruction Following, and Depth Quality, and validate its alignment with human judgments. Experiments with our developed multi-agent system DR^{3}-Agent based on multiple state-of-the-art language models demonstrate that DR^{3}-Eval is highly challenging and reveals critical failure modes in retrieval robustness and hallucination control. Our code and data are publicly available.
DR$^{3}$-Eval is a benchmark for evaluating deep research agents on multimodal, multi-file report generation, featuring a realistic simulation of web environments and a comprehensive evaluation framework.
deep research agents multimodal understanding report generation research sandbox corpus multi-dimensional evaluation framework information recall factual accuracy citation coverage instruction following depth quality hallucination control multi-agent system state-of-the-art language models