InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 45% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.
69e041dd5b704445ba50b64f
Oliver Bentham
69e041dd5b704445ba50b650
Vivek Srikumar
InfiniteScienceGym: An Unbounded, Procedurally-Generated Benchmark for Scientific Analysis
Large language models are emerging as scientific assistants, but evaluating their ability to reason from empirical data remains challenging. Benchmarks derived from published studies and human annotations inherit publication bias, known-knowledge bias, label noise, and substantial storage requirements. We present InfiniteScienceGym, a procedurally generated benchmark of scientific repositories paired with a verifiable question-answering task. From a seed, the simulator deterministically generates a self-contained repository with realistic directory structure, files, and tabular data, and a privileged QA generator produces both answerable and unanswerable questions with exact ground truth. This makes it possible to evaluate evidence-grounded reasoning, abstention, and tool-mediated analysis in a controlled setting without distributing a large static corpus. InfiniteScienceGym complements real scientific benchmarks by targeting blind spots and failure modes that are hard to evaluate using published datasets alone. Evaluating both proprietary and open-weight models, we find that none achieve more than 45% accuracy overall, that recognizing unanswerable questions remains a major weakness, and that stronger models tend to use tools more effectively rather than simply consuming more tokens.
InfiniteScienceGym presents a procedurally generated benchmark for evaluating scientific reasoning in language models, addressing limitations of traditional benchmarks through deterministic repository simulation and verifiable question-answering tasks.
large language models scientific assistants empirical data benchmarks publication bias known-knowledge bias label noise evidence-grounded reasoning abstention tool-mediated analysis procedural generation scientific repositories question-answering task deterministic simulation self-contained repository tabular data privileged QA generator ground truth controlled setting blind spots failure modes proprietary models open-weight models accuracy token consumption