Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles
Deep Research Agents (DRAs) have demonstrated remarkable capabilities in autonomous information retrieval and report generation, showing great potential to assist humans in complex research tasks. Current evaluation frameworks primarily rely on LLM-generated references or LLM-derived evaluation dimensions. While these approaches offer scalability, they often lack the reliability of expert-verified content and struggle to provide objective, fine-grained assessments of critical dimensions. To bridge this gap, we introduce Wiki Live Challenge (WLC), a live benchmark that leverages the newest Wikipedia Good Articles (GAs) as expert-level references. Wikipedia's strict standards for neutrality, comprehensiveness, and verifiability serve as a great challenge for DRAs, with GAs representing the pinnacle of which. We curate a dataset of 100 recent Good Articles and propose Wiki Eval, a comprehensive evaluation framework comprising a fine-grained evaluation method with 39 criteria for writing quality and rigorous metrics for factual verifiability. Extensive experiments on various DRA systems demonstrate a significant gap between current DRAs and human expert-level Wikipedia articles, validating the effectiveness of WLC in advancing agent research. We release our benchmark at https://github.com/WangShao2000/Wiki_Live_Challenge
69817335ce18b1862809603c
Shaohan Wang
claimed_verified
2026-02-03T10:07:30.979Z
6440d49b7663594a126716f2
/avatars/bb04af9d1ae9c5bc058ebfbf08f4ebc8.svg
shaohanwang
WShao
user
WShao
69817335ce18b1862809603d
Benfeng Xu
69817335ce18b1862809603e
Licheng Zhang
69817335ce18b1862809603f
Mingxuan Du
claimed_verified
2026-02-03T10:07:27.339Z
646dbba74ad7f907279dd486
https://cdn-avatars.huggingface.co/v1/production/uploads/646dbba74ad7f907279dd486/XGHJMFEIpWeDlh0cn9Slu.png
Mingxuan Du
Ayanami0730
user
Ayanami0730
69817335ce18b18628096040
Chiwei Zhu
claimed_verified
2026-02-03T10:07:25.343Z
663b22a80966eef8686aadaf
https://cdn-avatars.huggingface.co/v1/production/uploads/663b22a80966eef8686aadaf/iBzyQTyGZKf33RPVIFh9a.jpeg
Chiwei Zhu
IgnoraZ
user
IgnoraZ
69817335ce18b18628096041
Xiaorui Wang
69817335ce18b18628096042
Zhendong Mao
69817335ce18b18628096043
Yongdong Zhang
Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles
Deep Research Agents (DRAs) have demonstrated remarkable capabilities in autonomous information retrieval and report generation, showing great potential to assist humans in complex research tasks. Current evaluation frameworks primarily rely on LLM-generated references or LLM-derived evaluation dimensions. While these approaches offer scalability, they often lack the reliability of expert-verified content and struggle to provide objective, fine-grained assessments of critical dimensions. To bridge this gap, we introduce Wiki Live Challenge (WLC), a live benchmark that leverages the newest Wikipedia Good Articles (GAs) as expert-level references. Wikipedia's strict standards for neutrality, comprehensiveness, and verifiability serve as a great challenge for DRAs, with GAs representing the pinnacle of which. We curate a dataset of 100 recent Good Articles and propose Wiki Eval, a comprehensive evaluation framework comprising a fine-grained evaluation method with 39 criteria for writing quality and rigorous metrics for factual verifiability. Extensive experiments on various DRA systems demonstrate a significant gap between current DRAs and human expert-level Wikipedia articles, validating the effectiveness of WLC in advancing agent research. We release our benchmark at https://github.com/WangShao2000/Wiki_Live_Challenge
Deep Research Agents demonstrate capabilities in autonomous information retrieval but show significant gaps when evaluated against expert-level Wikipedia articles using a new live benchmark and comprehensive evaluation framework.
Deep Research Agents Wikipedia Good Articles live benchmark evaluation framework fine-grained evaluation factual verifiability agent research