ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert-written, fine-grained rubrics to assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop human and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state-of-the-art DR systems and find that even leading agents like Gemini's DR and OpenAI's DR achieve under 68% average compliance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release ResearchRubrics(including all prompts, rubrics, and evaluation code) to facilitate progress toward well-justified research assistants.
691708a2b63bfc66e0498739
Manasi Sharma
claimed_verified
2025-11-17T10:31:15.500Z
6912801e7487825c90d8e5eb
/avatars/2d9a6c83ba5c4b7b6276f8fa266a6307.svg
Manasi Sharma
manasisharma
user
manasisharma
691708a2b63bfc66e049873a
Chen Bo Calvin Zhang
691708a2b63bfc66e049873b
Chaithanya Bandi
691708a2b63bfc66e049873c
Clinton Wang
691708a2b63bfc66e049873d
Ankit Aich
691708a2b63bfc66e049873e
Huy Nghiem
691708a2b63bfc66e049873f
Tahseen Rabbani
691708a2b63bfc66e0498740
Ye Htet
691708a2b63bfc66e0498741
Brian Jang
691708a2b63bfc66e0498742
Sumana Basu
691708a2b63bfc66e0498743
Aishwarya Balwani
691708a2b63bfc66e0498744
Denis Peskoff
691708a2b63bfc66e0498745
Marcos Ayestaran
691708a2b63bfc66e0498746
Sean M. Hendryx
691708a2b63bfc66e0498747
Brad Kenstler
691708a2b63bfc66e0498748
Bing Liu
ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents
Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because responses are lengthy and diverse, admit many valid solutions, and often depend on dynamic information sources. We introduce ResearchRubrics, a standardized benchmark for DR built with over 2,800+ hours of human labor that pairs realistic, domain-diverse prompts with 2,500+ expert-written, fine-grained rubrics to assess factual grounding, reasoning soundness, and clarity. We also propose a new complexity framework for categorizing DR tasks along three axes: conceptual breadth, logical nesting, and exploration. In addition, we develop human and model-based evaluation protocols that measure rubric adherence for DR agents. We evaluate several state-of-the-art DR systems and find that even leading agents like Gemini's DR and OpenAI's DR achieve under 68% average compliance with our rubrics, primarily due to missed implicit context and inadequate reasoning about retrieved information. Our results highlight the need for robust, scalable assessment of deep research capabilities, to which end we release ResearchRubrics(including all prompts, rubrics, and evaluation code) to facilitate progress toward well-justified research assistants.
ResearchRubrics is a benchmark for evaluating deep research agents, using expert rubrics to assess their factual grounding, reasoning, and clarity across diverse, complex tasks.
large language models multi-step reasoning cross-document synthesis evidence-backed long-form answers ResearchRubrics factual grounding reasoning soundness clarity complexity framework conceptual breadth logical nesting exploration rubric adherence Gemini's DR OpenAI's DR