Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.
69b3738f2ad6c5d4cf4869f4
Łukasz Borchmann
admin_assigned
2026-03-13T08:59:21.102Z
600b381d3cc3b87db94bc0ce
https://cdn-avatars.huggingface.co/v1/production/uploads/600b381d3cc3b87db94bc0ce/I3xpr4gzcG1uXawXBpWpD.jpeg
Łukasz Borchmann
Borchmann
user
Borchmann
69b3738f2ad6c5d4cf4869f5
Jordy Van Landeghem
admin_assigned
2026-03-13T08:59:27.100Z
60001d6e556511aed81fd6b7
https://cdn-avatars.huggingface.co/v1/production/uploads/1654072894949-60001d6e556511aed81fd6b7.jpeg
Jordy Van Landeghem
jordyvl
user
jordyvl
69b3738f2ad6c5d4cf4869f6
Michał Turski
admin_assigned
2026-03-13T08:59:32.772Z
680268a7fd1fae58d58a2b49
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/P96qvBdInKYnVxij7-MM_.png
Michał Turski
mturski
user
mturski
69b3738f2ad6c5d4cf4869f7
Shreyansh Padarha
claimed_verified
2026-03-13T08:12:58.084Z
67ddd1b28c7f8d08d905b3f8
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/tCGZ__fi6vXtvg3rv0N7t.png
Shreyansh Padarha
shreyanshpadarha
user
shreyanshpadarha
69b3738f2ad6c5d4cf4869f8
Ryan Othniel Kearns
claimed_verified
2026-03-16T10:43:50.351Z
6751ea855b1cc8f5a14dcc27
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Vw5XZptgrzw2en-xQekAs.png
Ryan Othniel Kearns
ryanothk
user
ryanothk
69b3738f2ad6c5d4cf4869f9
Adam Mahdi
admin_assigned
2026-03-13T08:59:43.862Z
673cf9c93af47d1d2b6584c9
https://cdn-avatars.huggingface.co/v1/production/uploads/673cf9c93af47d1d2b6584c9/SmxAQGOgn35RwgqFbKbdL.jpeg
Adam Mahdi
ammaox
user
ammaox
69b3738f2ad6c5d4cf4869fa
Niels Rogge
claimed_verified
2026-03-13T08:51:40.989Z
5f1158120c833276f61f1a84
https://cdn-avatars.huggingface.co/v1/production/uploads/1608042047613-5f1158120c833276f61f1a84.jpeg
Niels Rogge
nielsr
user
nielsr
69b3738f2ad6c5d4cf4869fb
Clémentine Fourrier
admin_assigned
2026-03-13T08:59:51.025Z
6202a599216215a22221dea9
https://cdn-avatars.huggingface.co/v1/production/uploads/1644340617257-noauth.png
Clémentine Fourrier
clefourrier
user
clefourrier
69b3738f2ad6c5d4cf4869fc
Siwei Han
69b3738f2ad6c5d4cf4869fd
Huaxiu Yao
claimed_verified
2026-03-19T13:03:53.324Z
65f667a455009c4ad9e6ac4c
/avatars/ed10c2cf2ba3fde6d7da93f076961607.svg
Yao
Huaxiu
user
Huaxiu
69b3738f2ad6c5d4cf4869fe
Artemis Llabrés
admin_assigned
2026-03-13T09:00:14.359Z
630f3e904dbbbb6f667eceb7
https://cdn-avatars.huggingface.co/v1/production/uploads/630f3e904dbbbb6f667eceb7/Zyzby4Ruo70wmNhmYC_v4.jpeg
Artemis Llabrés
Llabres
user
Llabres
69b3738f2ad6c5d4cf4869ff
Yiming Xu
69b3738f2ad6c5d4cf486a00
Dimosthenis Karatzas
admin_assigned
2026-03-13T09:00:26.143Z
67c99d9405e0d2d1ed426597
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/2CTtTM6nuj2OMzyXMOOQd.png
Dimosthenis Karatzas
dkaratzas
user
dkaratzas
69b3738f2ad6c5d4cf486a01
Hao Zhang
69b3738f2ad6c5d4cf486a02
Anupam Datta
Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections
Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning.
MADQA benchmark evaluates multimodal agents' strategic reasoning capabilities through diverse PDF document questions, revealing gaps between human-level accuracy and efficient reasoning performance.
Multimodal agents document-intensive workflows strategic reasoning Classical Test Theory accuracy-effort trade-off brute-force search oracle performance