Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

States Hidden in Hidden States: LLMs Emerge Discrete State Representations Implicitly

Junhao Chen, Shengding Hu, Zhiyuan Liu, Maosong Sun

TL;DR

This work tackles how LLMs can implicitly perform extended arithmetic without step-by-step reasoning by hypothesizing Implicit Discrete State Representations (IDSRs) hidden in model states. Using a synthetic dataset of consecutive additions and probing with multi-layer perceptrons, the authors demonstrate IDSR existence, characterize their digit-, sequence-, and layer-level formation, and show that IDSRs are employed to deliver final results. They reveal a dual-layer regime: the first ten layers (shallow-semantic) generate arithmetic content with high fidelity, while later layers (semantic) re-encode information under task context, enabling multi-hop calculations via an attention-bridge mechanism. The findings advance interpretability by showing how mid- to long-range computations may be anchored in discrete internal states, though current open-source models still lose some information across steps, offering avenues to improve reliability and scalability of implicit computation in LLMs.

Abstract

Large Language Models (LLMs) exhibit various emergent abilities. Among these abilities, some might reveal the internal working mechanisms of models. In this paper, we uncover a novel emergent capability in models: the intrinsic ability to perform extended sequences of calculations without relying on chain-of-thought step-by-step solutions. Remarkably, the most advanced models can directly output the results of two-digit number additions with lengths extending up to 15 addends. We hypothesize that the model emerges Implicit Discrete State Representations (IDSRs) within its hidden states and performs symbolic calculations internally. To test this hypothesis, we design a sequence of experiments that look into the hidden states. Specifically, we first confirm that IDSRs exist. Then, we provide interesting observations about the formation of IDSRs from layer, digit, and sequence perspectives. Finally, we confirm that models indeed use IDSRs to produce the final answers. However, we also discover that these state representations are far from lossless in current open-sourced models, leading to inaccuracies in their final performance. Our work presents a novel exploration of LLMs' symbolic calculation abilities and the underlying mechanisms.

States Hidden in Hidden States: LLMs Emerge Discrete State Representations Implicitly

TL;DR

This work tackles how LLMs can implicitly perform extended arithmetic without step-by-step reasoning by hypothesizing Implicit Discrete State Representations (IDSRs) hidden in model states. Using a synthetic dataset of consecutive additions and probing with multi-layer perceptrons, the authors demonstrate IDSR existence, characterize their digit-, sequence-, and layer-level formation, and show that IDSRs are employed to deliver final results. They reveal a dual-layer regime: the first ten layers (shallow-semantic) generate arithmetic content with high fidelity, while later layers (semantic) re-encode information under task context, enabling multi-hop calculations via an attention-bridge mechanism. The findings advance interpretability by showing how mid- to long-range computations may be anchored in discrete internal states, though current open-source models still lose some information across steps, offering avenues to improve reliability and scalability of implicit computation in LLMs.

Abstract

Large Language Models (LLMs) exhibit various emergent abilities. Among these abilities, some might reveal the internal working mechanisms of models. In this paper, we uncover a novel emergent capability in models: the intrinsic ability to perform extended sequences of calculations without relying on chain-of-thought step-by-step solutions. Remarkably, the most advanced models can directly output the results of two-digit number additions with lengths extending up to 15 addends. We hypothesize that the model emerges Implicit Discrete State Representations (IDSRs) within its hidden states and performs symbolic calculations internally. To test this hypothesis, we design a sequence of experiments that look into the hidden states. Specifically, we first confirm that IDSRs exist. Then, we provide interesting observations about the formation of IDSRs from layer, digit, and sequence perspectives. Finally, we confirm that models indeed use IDSRs to produce the final answers. However, we also discover that these state representations are far from lossless in current open-sourced models, leading to inaccuracies in their final performance. Our work presents a novel exploration of LLMs' symbolic calculation abilities and the underlying mechanisms.
Paper Structure (32 sections, 2 equations, 11 figures, 3 tables)

This paper contains 32 sections, 2 equations, 11 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs' State Tracking Abilities.
  4. Interpretability of LLMs' Arithmetic Abilities.
  5. Broader Interpretability of LLMs.
  6. Emergence of Implicit Computation
  7. Problem Statement.
  8. Analysis Methodology
  9. Experiment Setup
  10. Dataset
  11. Hidden States
  12. Classification Probes
  13. Metrics
  14. Models Chosen
  15. Existence and Properties of IDSRs
  16. ...and 17 more sections

Figures (11)

  • Figure 1: Accuracies of Different Models Performing Consecutive 2-Digit Addition
  • Figure 2: Accuracies of Different Models Performing 10 Addend 2-Digit Addition
  • Figure 3: Accuracies of Qwen Series Performing Consecutive 2-Digit Addition
  • Figure 4: Accuracies of Whole Number Probing Predictions
  • Figure 5: Accuracies of By-Digit Probing Predictions
  • ...and 6 more figures