RiddleBench: A New Generative Reasoning Benchmark for LLMs
Large Language Models have demonstrated strong performance on many
established reasoning benchmarks. However, these benchmarks primarily evaluate
structured skills like quantitative problem-solving, leaving a gap in assessing
flexible, multifaceted reasoning abilities that are central to human
intelligence. These abilities require integrating logical deduction with
spatial awareness and constraint satisfaction, which current evaluations do not
measure well. To address this, we introduce RiddleBench, a benchmark of 1,737
challenging puzzles in English designed to probe these core reasoning
capabilities. Evaluation of state-of-the-art models on RiddleBench shows
fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3,
and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and
63.16%). Analysis further reveals deep failures, including hallucination
cascades (accepting flawed reasoning from other models) and poor
self-correction due to a strong self-confirmation bias. Their reasoning is also
fragile, with performance degrading significantly when constraints are
reordered or irrelevant information is introduced. RiddleBench functions as a
diagnostic tool for these issues and as a resource for guiding the development
of more robust and reliable language models.
6908db77812eca10f9cc6095
Deepon Halder
claimed_verified
2025-11-05T16:25:09.006Z
66fc42b48bf64649c684b479
https://cdn-avatars.huggingface.co/v1/production/uploads/66fc42b48bf64649c684b479/7ja7AWVPELZmHMUmIhaPP.jpeg
Deepon
deeponh
user
deeponh
6908db77812eca10f9cc6096
Alan Saji
6908db77812eca10f9cc6097
Thanmay Jayakumar
6908db77812eca10f9cc6098
Ratish Puduppully
6908db77812eca10f9cc6099
Anoop Kunchukuttan
6908db77812eca10f9cc609a
Raj Dabre
RiddleBench: A New Generative Reasoning Benchmark for LLMs
Large Language Models have demonstrated strong performance on many
established reasoning benchmarks. However, these benchmarks primarily evaluate
structured skills like quantitative problem-solving, leaving a gap in assessing
flexible, multifaceted reasoning abilities that are central to human
intelligence. These abilities require integrating logical deduction with
spatial awareness and constraint satisfaction, which current evaluations do not
measure well. To address this, we introduce RiddleBench, a benchmark of 1,737
challenging puzzles in English designed to probe these core reasoning
capabilities. Evaluation of state-of-the-art models on RiddleBench shows
fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3,
and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and
63.16%). Analysis further reveals deep failures, including hallucination
cascades (accepting flawed reasoning from other models) and poor
self-correction due to a strong self-confirmation bias. Their reasoning is also
fragile, with performance degrading significantly when constraints are
reordered or irrelevant information is introduced. RiddleBench functions as a
diagnostic tool for these issues and as a resource for guiding the development
of more robust and reliable language models.
RiddleBench, a benchmark of 1,737 puzzles, reveals fundamental weaknesses in state-of-the-art language models, including hallucination cascades and poor self-correction, highlighting the need for more robust reasoning capabilities.
Large Language Models reasoning benchmarks structured skills quantitative problem-solving flexible reasoning multifaceted reasoning logical deduction spatial awareness constraint satisfaction RiddleBench hallucination cascades self-confirmation bias