Sparse Reward Subsystem in Large Language Models
In this paper, we identify a sparse reward subsystem within the hidden states of Large Language Models (LLMs), drawing an analogy to the biological reward subsystem in the human brain. We demonstrate that this subsystem contains value neurons that represent the model's internal expectation of state value, and through intervention experiments, we establish the importance of these neurons for reasoning. Our experiments reveal that these value neurons are robust across diverse datasets, model scales, and architectures; furthermore, they exhibit significant transferability across different datasets and models fine-tuned from the same base model. By examining cases where value predictions and actual rewards diverge, we identify dopamine neurons within the reward subsystem which encode reward prediction errors (RPE). These neurons exhibit high activation when the reward is higher than expected and low activation when the reward is lower than expected.
698212609c2f139721ec34c1
Guowei Xu
claimed_verified
2026-02-04T12:31:21.969Z
63e992cdccae1fe5c6222f84
https://cdn-avatars.huggingface.co/v1/production/uploads/63e992cdccae1fe5c6222f84/IvksSUf2DENfUwCZSmNPd.jpeg
Guowei Xu
Xkev
user
Xkev
698212609c2f139721ec34c2
Mert Yuksekgonul
698212609c2f139721ec34c3
James Zou
Sparse Reward Subsystem in Large Language Models
In this paper, we identify a sparse reward subsystem within the hidden states of Large Language Models (LLMs), drawing an analogy to the biological reward subsystem in the human brain. We demonstrate that this subsystem contains value neurons that represent the model's internal expectation of state value, and through intervention experiments, we establish the importance of these neurons for reasoning. Our experiments reveal that these value neurons are robust across diverse datasets, model scales, and architectures; furthermore, they exhibit significant transferability across different datasets and models fine-tuned from the same base model. By examining cases where value predictions and actual rewards diverge, we identify dopamine neurons within the reward subsystem which encode reward prediction errors (RPE). These neurons exhibit high activation when the reward is higher than expected and low activation when the reward is lower than expected.
Research identifies a sparse reward subsystem in LLM hidden states containing value neurons that represent internal state expectations and dopamine-like neurons encoding reward prediction errors.
Large Language Models reward subsystem value neurons reward prediction errors dopamine neurons hidden states intervention experiments transferability neural representation