MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome
Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.
69cdda45acf91530b21df965
Fangda Ye
69cdda45acf91530b21df966
Yuxin Hu
69cdda45acf91530b21df967
Pengxiang Zhu
claimed_verified
2026-04-02T15:29:55.435Z
65d447b24c1bd7a42f225501
/avatars/44bb36cd547528fb8f2850ab798f67f0.svg
Pengxiang Zhu
JubSteven
user
JubSteven
69cdda45acf91530b21df968
Yibo Li
claimed_verified
2026-04-02T08:13:15.790Z
6569474501c02495cec2cbae
/avatars/5d383738269a092192f3822d0248fd43.svg
Yibo Li
liushiliushi
user
liushiliushi
69cdda45acf91530b21df969
Ziqi Jin
claimed_verified
2026-04-09T08:07:50.376Z
625921d05f80a3c1aad0bae3
https://cdn-avatars.huggingface.co/v1/production/uploads/625921d05f80a3c1aad0bae3/ElN3-6V5nGId2fzI3Dqlr.jpeg
Phi
Xalphinions
user
Xalphinions
69cdda45acf91530b21df96a
Yao Xiao
69cdda45acf91530b21df96b
Yibo Wang
69cdda45acf91530b21df96c
Lei Wang
69cdda45acf91530b21df96d
Zhen Zhang
69cdda45acf91530b21df96e
Lu Wang
69cdda45acf91530b21df96f
Yue Deng
69cdda45acf91530b21df970
Bin Wang
69cdda45acf91530b21df971
Yifan Zhang
69cdda45acf91530b21df972
Liangcai Su
69cdda45acf91530b21df973
Xinyu Wang
69cdda45acf91530b21df974
He Zhao
69cdda45acf91530b21df975
Chen Wei
69cdda45acf91530b21df976
Qiang Ren
69cdda45acf91530b21df977
Bryan Hooi
69cdda45acf91530b21df978
An Bo
69cdda45acf91530b21df979
Shuicheng Yan
69cdda45acf91530b21df97a
Lidong Bing
MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome
Recent progress in deep research systems has been impressive, but evaluation still lags behind real user needs. Existing benchmarks predominantly assess final reports using fixed rubrics, failing to evaluate the underlying research process. Most also offer limited multimodal coverage, rely on synthetic tasks that do not reflect real-world query complexity, and cannot be refreshed as knowledge evolves. To address these gaps, we introduce MiroEval, a benchmark and evaluation framework for deep research systems. The benchmark comprises 100 tasks (70 text-only, 30 multimodal), all grounded in real user needs and constructed via a dual-path pipeline that supports periodic updates, enabling a live and evolving setting. The proposed evaluation suite assesses deep research systems along three complementary dimensions: adaptive synthesis quality evaluation with task-specific rubrics, agentic factuality verification via active retrieval and reasoning over both web sources and multimodal attachments, and process-centric evaluation audits how the system searches, reasons, and refines throughout its investigation. Evaluation across 13 systems yields three principal findings: the three evaluation dimensions capture complementary aspects of system capability, with each revealing distinct strengths and weaknesses across systems; process quality serves as a reliable predictor of overall outcome while revealing weaknesses invisible to output-level metrics; and multimodal tasks pose substantially greater challenges, with most systems declining by 3 to 10 points. The MiroThinker series achieves the most balanced performance, with MiroThinker-H1 ranking the highest overall in both settings. Human verification and robustness results confirm the reliability of the benchmark and evaluation framework. MiroEval provides a holistic diagnostic tool for the next generation of deep research agents.
MiroEval addresses limitations of existing deep research system benchmarks by introducing a comprehensive evaluation framework that assesses adaptive synthesis, agentic factuality verification, and process-centric auditing across real-user tasks.
deep research systems evaluation framework adaptive synthesis quality evaluation agentic factuality verification process-centric evaluation multimodal coverage real user needs dual-path pipeline live evolving setting deep research agents