Robust Reasoning Benchmark
While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their underlying reasoning processes remain highly overfit to standard textual formatting. We propose a perturbation pipeline consisting of 14 techniques to evaluate robustness of LLM reasoning. We apply this pipeline to AIME 2024 dataset and evalute 8 state-of-the-art models on the resulting benchmark. While frontier models exhibit resilience, open weights reasoning models suffer catastrophic collapses (up to 55% average accuracy drops across perturbations and up to 100% on some), exposing structural fragility. To further disentangle mechanical parsing failures from downstream reasoning failures, we strictly isolate the models' working memory capacity by forcing models to solve multiple unperturbed mathematical problems sequentially within a single context window. Our results indicate that open weight models ranging from 7B to 120B parameters and Claude Opus 4.6 exhibit accuracy decay on subsequent problems. This degradation demonstrates that intermediate reasoning steps permanently pollute standard dense attention mechanisms. We argue that to achieve reliable reasoning, future reasoning architectures must integrate explicit contextual resets within a model's own Chain-of-Thought, leading to fundamental open questions regarding the optimal granularity of atomic reasoning tasks.
69dc4d8e2b6b8dc66ffeebac
Pavel Golikov
claimed_verified
2026-04-13T07:32:26.823Z
65ce8d0e52e9d72c908e57d3
/avatars/5efa04771596f487f6c285875d8ff3c0.svg
Golikov
PavelGolikov
user
PavelGolikov
69dc4d8e2b6b8dc66ffeebad
Evgenii Opryshko
69dc4d8e2b6b8dc66ffeebae
Gennady Pekhimenko
69dc4d8e2b6b8dc66ffeebaf
Mark C. Jeffrey
Robust Reasoning Benchmark
While Large Language Models (LLMs) achieve high performance on standard mathematical benchmarks, their underlying reasoning processes remain highly overfit to standard textual formatting. We propose a perturbation pipeline consisting of 14 techniques to evaluate robustness of LLM reasoning. We apply this pipeline to AIME 2024 dataset and evalute 8 state-of-the-art models on the resulting benchmark. While frontier models exhibit resilience, open weights reasoning models suffer catastrophic collapses (up to 55% average accuracy drops across perturbations and up to 100% on some), exposing structural fragility. To further disentangle mechanical parsing failures from downstream reasoning failures, we strictly isolate the models' working memory capacity by forcing models to solve multiple unperturbed mathematical problems sequentially within a single context window. Our results indicate that open weight models ranging from 7B to 120B parameters and Claude Opus 4.6 exhibit accuracy decay on subsequent problems. This degradation demonstrates that intermediate reasoning steps permanently pollute standard dense attention mechanisms. We argue that to achieve reliable reasoning, future reasoning architectures must integrate explicit contextual resets within a model's own Chain-of-Thought, leading to fundamental open questions regarding the optimal granularity of atomic reasoning tasks.
Research reveals that large language models exhibit fragile reasoning capabilities when subjected to perturbations, with open-weight models showing significant accuracy drops and evidence of memory pollution in dense attention mechanisms.
Large Language Models mathematical benchmarks perturbation pipeline AIME 2024 dataset state-of-the-art models reasoning processes contextual resets Chain-of-Thought dense attention mechanisms working memory capacity