Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy
Large language models (LLMs), despite strong performance on complex mathematical problems, exhibit systematic limitations in counting tasks. This issue arises from architectural limits of transformers, where counting is performed across layers, leading to degraded precision for larger counting problems due to depth constraints. To address this limitation, we propose a simple test-time strategy inspired by System-2 cognitive processes that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. We evaluate this approach using observational and causal mediation analyses to understand the underlying mechanism of this System-2-like strategy. Our mechanistic analysis identifies key components: latent counts are computed and stored in the final item representations of each part, transferred to intermediate steps via dedicated attention heads, and aggregated in the final stage to produce the total count. Experimental results demonstrate that this strategy enables LLMs to surpass architectural limitations and achieve high accuracy on large-scale counting tasks. This work provides mechanistic insight into System-2 counting in LLMs and presents a generalizable approach for improving and understanding their reasoning behavior.
695e4ecfc679bee50ca69b4e
Hosein Hasani
695e4ecfc679bee50ca69b4f
Mohammadali Banayeeanzade
695e4ecfc679bee50ca69b50
Ali Nafisi
claimed_verified
2026-01-07T13:13:16.135Z
65aa7c6cc8903e28aea576e7
https://cdn-avatars.huggingface.co/v1/production/uploads/65aa7c6cc8903e28aea576e7/1jcif5QJ2dKu_oeBNVqYG.jpeg
Ali Nafisi
safinal
user
safinal
695e4ecfc679bee50ca69b51
Sadegh Mohammadian
695e4ecfc679bee50ca69b52
Fatemeh Askari
695e4ecfc679bee50ca69b53
Mobin Bagherian
695e4ecfc679bee50ca69b54
Amirmohammad Izadi
695e4ecfc679bee50ca69b55
Mahdieh Soleymani Baghshah
Mechanistic Interpretability of Large-Scale Counting in LLMs through a System-2 Strategy
Large language models (LLMs), despite strong performance on complex mathematical problems, exhibit systematic limitations in counting tasks. This issue arises from architectural limits of transformers, where counting is performed across layers, leading to degraded precision for larger counting problems due to depth constraints. To address this limitation, we propose a simple test-time strategy inspired by System-2 cognitive processes that decomposes large counting tasks into smaller, independent sub-problems that the model can reliably solve. We evaluate this approach using observational and causal mediation analyses to understand the underlying mechanism of this System-2-like strategy. Our mechanistic analysis identifies key components: latent counts are computed and stored in the final item representations of each part, transferred to intermediate steps via dedicated attention heads, and aggregated in the final stage to produce the total count. Experimental results demonstrate that this strategy enables LLMs to surpass architectural limitations and achieve high accuracy on large-scale counting tasks. This work provides mechanistic insight into System-2 counting in LLMs and presents a generalizable approach for improving and understanding their reasoning behavior.
A test-time strategy inspired by System-2 cognitive processes decomposes large counting tasks into smaller sub-problems, enabling large language models to overcome architectural limitations and achieve high accuracy on complex counting tasks through mechanistic components like latent count computation, dedicated attention heads, and final aggregation stages.
large language models transformers counting tasks System-2 cognitive processes test-time strategy latent counts attention heads mechanistic analysis causal mediation analysis