OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.
69af9608b3ab31dee1281d98
Krista Opsahl-Ong
69af9608b3ab31dee1281d99
Arnav Singhvi
69af9608b3ab31dee1281d9a
Jasmine Collins
69af9608b3ab31dee1281d9b
Ivan Zhou
69af9608b3ab31dee1281d9c
Cindy Wang
69af9608b3ab31dee1281d9d
Ashutosh Baheti
69af9608b3ab31dee1281d9e
Owen Oertell
69af9608b3ab31dee1281d9f
Jacob Portes
69af9608b3ab31dee1281da0
Sam Havens
69af9608b3ab31dee1281da1
Erich Elsen
69af9608b3ab31dee1281da2
Michael Bendersky
69af9608b3ab31dee1281da3
Matei Zaharia
69af9608b3ab31dee1281da4
Xing Chen
OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning
We introduce OfficeQA Pro, a benchmark for evaluating AI agents on grounded, multi-document reasoning over a large and heterogeneous document corpus. The corpus consists of U.S. Treasury Bulletins spanning nearly 100 years, comprising 89,000 pages and over 26 million numerical values. OfficeQA Pro consists of 133 questions that require precise document parsing, retrieval, and analytical reasoning across both unstructured text and tabular data. Frontier LLMs including Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro Preview achieve less than 5% accuracy on OfficeQA Pro when relying on parametric knowledge, and less than 12% with additional access to the web. When provided directly with the document corpus, frontier agents still struggle on over half of questions, scoring 34.1% on average. We find that providing agents with a structured document representation produced by Databricks' ai_parse_document yields a 16.1% average relative performance gain across agents. We conduct additional ablations to study the effects of model selection, table representation, retrieval strategy, and test-time scaling on performance. Despite these improvements, significant headroom remains before agents can be considered reliable at enterprise-grade grounded reasoning.
OfficeQA Pro evaluates AI agents on multi-document reasoning across historical financial documents, revealing persistent challenges in grounded reasoning despite advanced model capabilities.
grounded reasoning multi-document reasoning document parsing retrieval analytical reasoning parametric knowledge web access structured document representation ai_parse_document test-time scaling