MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the decision-critical factors driving the model's behavior, leading to the reduced CoT monitorability problem. However, a comprehensive and fully open-source benchmark for studying CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments across multiple popular LLMs with varying capabilities show that CoT monitorability is higher when producing the final target response requires structural reasoning through the decision-critical factor. Closed-source LLMs generally show lower monitorability, and there exists a negative relationship between monitorability and model capability. Moreover, both open- and closed-source LLMs can intentionally reduce monitorability under stress-tests, with monitorability dropping by up to 30% in some tasks that do not require structural reasoning over the decision-critical factors. Beyond these empirical insights, MonitorBench provides a basis for further research on evaluating future LLMs, studying advanced stress-test monitorability techniques, and developing new monitoring approaches.
69cc30ed3f84a0971be66a06
Han Wang
claimed_verified
2026-04-01T08:35:09.594Z
64ce5c4a132efbe2dca17606
/avatars/c54d620d529498752081cad55f9a14a8.svg
Han Wang
rookiehabc
user
rookiehabc
69cc30ed3f84a0971be66a07
Yifan Sun
69cc30ed3f84a0971be66a08
Brian Ko
69cc30ed3f84a0971be66a09
Mann Talati
claimed_verified
2026-04-07T08:47:06.327Z
682797d505b4f3833dcf3908
/avatars/9e3156f63ca6f1240956b92c1d9fd25e.svg
Mann Talati
manntalati
user
manntalati
69cc30ed3f84a0971be66a0a
Jiawen Gong
69cc30ed3f84a0971be66a0b
Zimeng Li
69cc30ed3f84a0971be66a0c
Naicheng Yu
69cc30ed3f84a0971be66a0d
Xucheng Yu
69cc30ed3f84a0971be66a0e
Wei Shen
69cc30ed3f84a0971be66a0f
Vedant Jolly
69cc30ed3f84a0971be66a10
Huan Zhang
MonitorBench: A Comprehensive Benchmark for Chain-of-Thought Monitorability in Large Language Models
Large language models (LLMs) can generate chains of thought (CoTs) that are not always causally responsible for their final outputs. When such a mismatch occurs, the CoT no longer faithfully reflects the decision-critical factors driving the model's behavior, leading to the reduced CoT monitorability problem. However, a comprehensive and fully open-source benchmark for studying CoT monitorability remains lacking. To address this gap, we propose MonitorBench, a systematic benchmark for evaluating CoT monitorability in LLMs. MonitorBench provides: (1) a diverse set of 1,514 test instances with carefully designed decision-critical factors across 19 tasks spanning 7 categories to characterize when CoTs can be used to monitor the factors driving LLM behavior; and (2) two stress-test settings to quantify the extent to which CoT monitorability can be degraded. Extensive experiments across multiple popular LLMs with varying capabilities show that CoT monitorability is higher when producing the final target response requires structural reasoning through the decision-critical factor. Closed-source LLMs generally show lower monitorability, and there exists a negative relationship between monitorability and model capability. Moreover, both open- and closed-source LLMs can intentionally reduce monitorability under stress-tests, with monitorability dropping by up to 30% in some tasks that do not require structural reasoning over the decision-critical factors. Beyond these empirical insights, MonitorBench provides a basis for further research on evaluating future LLMs, studying advanced stress-test monitorability techniques, and developing new monitoring approaches.
MonitorBench is introduced as a comprehensive benchmark for evaluating chains of thought monitorability in large language models, revealing that monitorability decreases when structural reasoning is not required and that both open- and closed-source models exhibit reduced monitorability under stress-testing conditions.
chains of thought monitorability large language models decision-critical factors stress-test settings structural reasoning