Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sparse Reward Subsystem in Large Language Models

Guowei Xu, Mert Yuksekgonul, James Zou

TL;DR

The paper identifies a sparse reward subsystem in Large Language Models (LLMs), consisting of value neurons that encode the model's internal state-value estimates and dopamine neurons that reflect reward prediction error (RPE). It introduces a simple two-layer MLP value probe trained with Temporal Difference (TD) learning to extract this signal from hidden states, and demonstrates that a tiny fraction of neurons (often <1%) suffices to predict value, with ablations showing causal importance for reasoning. The authors show robust presence and transferability of these neurons across datasets, model scales, layers, and architectures, and establish cross-dataset and cross-model consistency via Intersection-over-Union analyses. They further identify dopamine neurons and reveal a functional link between value and dopamine neurons through ablation studies, suggesting these signals are intrinsic to the model's reasoning process and potentially useful for confidence estimation and compute allocation. Overall, the work provides a mechanistic view of intrinsic reward signals in LLMs and points to practical applications in monitoring and guiding generation, with implications for interpretability and safe deployment.

Abstract

In this paper, we identify a sparse reward subsystem within the hidden states of Large Language Models (LLMs), drawing an analogy to the biological reward subsystem in the human brain. We demonstrate that this subsystem contains value neurons that represent the model's internal expectation of state value, and through intervention experiments, we establish the importance of these neurons for reasoning. Our experiments reveal that these value neurons are robust across diverse datasets, model scales, and architectures; furthermore, they exhibit significant transferability across different datasets and models fine-tuned from the same base model. By examining cases where value predictions and actual rewards diverge, we identify dopamine neurons within the reward subsystem which encode reward prediction errors (RPE). These neurons exhibit high activation when the reward is higher than expected and low activation when the reward is lower than expected.

Sparse Reward Subsystem in Large Language Models

TL;DR

The paper identifies a sparse reward subsystem in Large Language Models (LLMs), consisting of value neurons that encode the model's internal state-value estimates and dopamine neurons that reflect reward prediction error (RPE). It introduces a simple two-layer MLP value probe trained with Temporal Difference (TD) learning to extract this signal from hidden states, and demonstrates that a tiny fraction of neurons (often <1%) suffices to predict value, with ablations showing causal importance for reasoning. The authors show robust presence and transferability of these neurons across datasets, model scales, layers, and architectures, and establish cross-dataset and cross-model consistency via Intersection-over-Union analyses. They further identify dopamine neurons and reveal a functional link between value and dopamine neurons through ablation studies, suggesting these signals are intrinsic to the model's reasoning process and potentially useful for confidence estimation and compute allocation. Overall, the work provides a mechanistic view of intrinsic reward signals in LLMs and points to practical applications in monitoring and guiding generation, with implications for interpretability and safe deployment.

Abstract

In this paper, we identify a sparse reward subsystem within the hidden states of Large Language Models (LLMs), drawing an analogy to the biological reward subsystem in the human brain. We demonstrate that this subsystem contains value neurons that represent the model's internal expectation of state value, and through intervention experiments, we establish the importance of these neurons for reasoning. Our experiments reveal that these value neurons are robust across diverse datasets, model scales, and architectures; furthermore, they exhibit significant transferability across different datasets and models fine-tuned from the same base model. By examining cases where value predictions and actual rewards diverge, we identify dopamine neurons within the reward subsystem which encode reward prediction errors (RPE). These neurons exhibit high activation when the reward is higher than expected and low activation when the reward is lower than expected.
Paper Structure (35 sections, 8 equations, 13 figures, 6 tables)

This paper contains 35 sections, 8 equations, 13 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Sparse Value Neurons in Large Language Models
  3. Training a Value Probe
  4. Identifying Value Neurons
  5. Empirical Evidence
  6. Intervention Experiments
  7. Robustness and Transferability of the Value Neurons
  8. Robustness on More Datasets
  9. Robustness across Different Model Scales
  10. Robustness across Different Layers
  11. Robustness across Different Models
  12. Transferability Across Different Datasets
  13. Transferability Across Fine-tuned Models
  14. Applications of the Value Neurons
  15. Identifying Dopamine Neurons
  16. ...and 20 more sections

Figures (13)

  • Figure 1: AUC curves for layers 2--4 of the Qwen-2.5-14B-SimpleRL-Zoo model on the GSM8K and MATH500 datasets. The curves indicate that the value probe can accurately predict the value by relying on only a very small number of value neurons.
  • Figure 2: AUC curves for layers 2--4 of the Qwen-2.5-14B-SimpleRL-Zoo model on the Minerva Math, ARC, and the STEM subset of MMLU datasets.
  • Figure 3: AUC curves for layers 2--4 of the Qwen-2.5-1.5B/7B/14B-SimpleRL-Zoo model on the GSM8K dataset.
  • Figure 4: AUC curves for different layers of the Qwen-2.5-1.5B-SimpleRL-Zoo model on the GSM8K dataset.
  • Figure 5: AUC curves for layers 2--4 of the Llama-3.1-8B-Instruct, Gemma-3-4B-it, and Phi-3.5-mini-instruct models on the MATH500 dataset.
  • ...and 8 more figures