PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error
  Detection
We introduce PRISM-Bench, a benchmark of puzzle-based visual
challenges designed to evaluate not only whether models can solve problems, but
how their reasoning unfolds. Unlike prior evaluations that measure only
final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual
puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error,
models must identify the first incorrect step. This setting enables
fine-grained assessment of logical consistency, error detection, and visual
reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric,
and analogical reasoning, resisting shortcuts based on superficial pattern
matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap
between fluent generation and faithful reasoning: models that produce plausible
CoTs often fail to locate simple logical faults. By disentangling answer
generation from reasoning verification, PRISM-Bench offers a sharper lens on
multimodal reasoning competence and underscores the need for diagnostic
evaluation protocols in the development of trustworthy MLLMs.
6900326e22d452aac6dd4314
Yusu Qian
6900326e22d452aac6dd4315
Cheng Wan
6900326e22d452aac6dd4316
Chao Jia
6900326e22d452aac6dd4317
Yinfei Yang
6900326e22d452aac6dd4318
Qingyu Zhao
6900326e22d452aac6dd4319
Zhe Gan
PRISM-Bench: A Benchmark of Puzzle-Based Visual Tasks with CoT Error
  Detection
We introduce PRISM-Bench, a benchmark of puzzle-based visual
challenges designed to evaluate not only whether models can solve problems, but
how their reasoning unfolds. Unlike prior evaluations that measure only
final-answer accuracy, PRISM-Bench introduces a diagnostic task: given a visual
puzzle and a step-by-step chain-of-thought (CoT) containing exactly one error,
models must identify the first incorrect step. This setting enables
fine-grained assessment of logical consistency, error detection, and visual
reasoning. The puzzles in PRISM-Bench require multi-step symbolic, geometric,
and analogical reasoning, resisting shortcuts based on superficial pattern
matching. Evaluations across state-of-the-art MLLMs reveal a persistent gap
between fluent generation and faithful reasoning: models that produce plausible
CoTs often fail to locate simple logical faults. By disentangling answer
generation from reasoning verification, PRISM-Bench offers a sharper lens on
multimodal reasoning competence and underscores the need for diagnostic
evaluation protocols in the development of trustworthy MLLMs.
PRISM-Bench evaluates models' reasoning processes by identifying errors in step-by-step solutions to visual puzzles, highlighting gaps between fluent generation and logical consistency.
PRISM-Bench chain-of-thought logical consistency error detection visual reasoning multi-step reasoning symbolic reasoning geometric reasoning analogical reasoning MLLMs multimodal reasoning