DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation
Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.
696864c90ac10a06522f6a4a
Yibo Wang
claimed_verified
2026-01-16T14:37:10.922Z
66f16d166f7038039a1e2770
/avatars/0a30d3e9af3b109ce4b82396b0e8d685.svg
Yibo Wang
yiboowang
user
yiboowang
696864c90ac10a06522f6a4b
Lei Wang
claimed_verified
2026-01-16T10:32:43.037Z
646def60df618b303b419323
https://cdn-avatars.huggingface.co/v1/production/uploads/646def60df618b303b419323/JLJGYen4-5M8ivsLsSk0w.jpeg
Lei Wang
demolei
user
demolei
696864c90ac10a06522f6a4c
Yue Deng
696864c90ac10a06522f6a4d
Keming Wu
claimed_verified
2026-01-15T15:02:22.232Z
66bf00ca5b4e241fe266059d
https://cdn-avatars.huggingface.co/v1/production/uploads/66bf00ca5b4e241fe266059d/VoWPC_C4zoeT6dS699t7L.png
Keming Wu
wukeming11
user
wukeming11
696864c90ac10a06522f6a4e
Yao Xiao
696864c90ac10a06522f6a4f
Huanjin Yao
696864c90ac10a06522f6a50
Liwei Kang
696864c90ac10a06522f6a51
Hai Ye
696864c90ac10a06522f6a52
Yongcheng Jing
696864c90ac10a06522f6a53
Lidong Bing
DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation
Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing.
DeepResearchEval presents an automated framework for creating complex research tasks and evaluating them through agent-based methods that adapt to task specifics and verify facts without relying on citations.
automated framework deep research task construction agentic evaluation persona-driven pipeline task qualification search necessity adaptive point-wise quality evaluation active fact-checking web search multi-source evidence integration