State over Tokens: Characterizing the Role of Reasoning Tokens
Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes, empirical evidence reveals that they are not a faithful explanation of the model's actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state -- the sole persistent information carrier across the model's stateless generation cycles. This explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We argue that to truly understand the process that LLMs do, research must move beyond reading the reasoning tokens as text and focus on decoding them as state.
69415bb065f1e24a11780895
Mosh Levy
69415bb065f1e24a11780896
Zohar Elyoseph
69415bb065f1e24a11780897
Shauli Ravfogel
69415bb065f1e24a11780898
Yoav Goldberg
State over Tokens: Characterizing the Role of Reasoning Tokens
Large Language Models (LLMs) can generate reasoning tokens before their final answer to boost performance on complex tasks. While these sequences seem like human thought processes, empirical evidence reveals that they are not a faithful explanation of the model's actual reasoning process. To address this gap between appearance and function, we introduce the State over Tokens (SoT) conceptual framework. SoT reframes reasoning tokens not as a linguistic narrative, but as an externalized computational state -- the sole persistent information carrier across the model's stateless generation cycles. This explains how the tokens can drive correct reasoning without being a faithful explanation when read as text and surfaces previously overlooked research questions on these tokens. We argue that to truly understand the process that LLMs do, research must move beyond reading the reasoning tokens as text and focus on decoding them as state.
The State over Tokens (SoT) framework reinterprets reasoning tokens in large language models as computational states rather than linguistic narratives, highlighting the need for a new focus in research.
Large Language Models (LLMs) reasoning tokens State over Tokens (SoT) computational state generation cycles