DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks that target single answer retrieval or broad-spectrum factuality, DeepSearchQA features a dataset of challenging, handcrafted tasks designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.
697c2f91a67238fac88cc159
Nikita Gupta
697c2f91a67238fac88cc15a
Riju Chatterjee
697c2f91a67238fac88cc15b
Lukas Haas
697c2f91a67238fac88cc15c
Connie Tao
697c2f91a67238fac88cc15d
Andrew Wang
697c2f91a67238fac88cc15e
Chang Liu
697c2f91a67238fac88cc15f
Hidekazu Oiwa
697c2f91a67238fac88cc160
Elena Gribovskaya
697c2f91a67238fac88cc161
Jan Ackermann
697c2f91a67238fac88cc162
John Blitzer
697c2f91a67238fac88cc163
Sasha Goldshtein
697c2f91a67238fac88cc164
Dipanjan Das
DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents
We introduce DeepSearchQA, a 900-prompt benchmark for evaluating agents on difficult multi-step information-seeking tasks across 17 different fields. Unlike traditional benchmarks that target single answer retrieval or broad-spectrum factuality, DeepSearchQA features a dataset of challenging, handcrafted tasks designed to evaluate an agent's ability to execute complex search plans to generate exhaustive answer lists. This shift in design explicitly tests three critical, yet under-evaluated capabilities: 1) systematic collation of fragmented information from disparate sources, 2) de-duplication and entity resolution to ensure precision, and 3) the ability to reason about stopping criteria within an open-ended search space. Each task is structured as a causal chain, where discovering information for one step is dependent on the successful completion of the previous one, stressing long-horizon planning and context retention. All tasks are grounded in the open web with objectively verifiable answer sets. Our comprehensive evaluation of state-of-the-art agent architectures reveals significant performance limitations: even the most advanced models struggle to balance high recall with precision. We observe distinct failure modes ranging from premature stopping (under-retrieval) to hedging behaviors, where agents cast an overly wide net of low-confidence answers to artificially boost recall. These findings highlight critical headroom in current agent designs and position DeepSearchQA as an essential diagnostic tool for driving future research toward more robust, deep-research capabilities.
DeepSearchQA presents a 900-prompt benchmark evaluating agents on complex multi-step information-seeking tasks requiring systematic information collation, deduplication, and reasoning about stopping criteria across 17 fields.
multi-step information-seeking tasks causal chain systematic collation de-duplication entity resolution long-horizon planning context retention open-ended search space agent architectures recall precision premature stopping hedging behaviors deep-research capabilities