Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic

Xingyu Zhao, Darsh Sharma, Rheeya Uppaal, Yiqiao Zhong

TL;DR

Transformers learning arithmetic subskills often deviate from human rule-based composition, exhibiting reverse and parallel acquisition that produces mixing errors, especially under distribution shifts. By training small transformers on synthetic tasks (addition, multiplication, comparison, sorting) and tracing subskill emergence with information-theoretic diagnostics, the authors show learning dynamics align with correlational signals in the data, not causal decompositions. These shattered compositionality behaviors persist across model scale, pretraining, and scratchpad reasoning, indicating a fundamental training-dynamics phenomenon with implications for reasoning reliability and robustness to out-of-distribution prompts. The work highlights the need for new evaluation and training strategies to mitigate brittle compositions in real-world LLM deployments and alignment contexts.

Abstract

Large language models (LLMs) often exhibit unexpected errors or unintended behavior, even at scale. While recent work reveals the discrepancy between LLMs and humans in skill compositions, the learning dynamics of skill compositions and the underlying cause of non-human behavior remain elusive. In this study, we investigate the mechanism of learning dynamics by training transformers on synthetic arithmetic tasks. Through extensive ablations and fine-grained diagnostic metrics, we discover that transformers do not reliably build skill compositions according to human-like sequential rules. Instead, they often acquire skills in reverse order or in parallel, which leads to unexpected mixing errors especially under distribution shifts--a phenomenon we refer to as shattered compositionality. To explain these behaviors, we provide evidence that correlational matching to the training data, rather than causal or procedural composition, shapes learning dynamics. We further show that shattered compositionality persists in modern LLMs and is not mitigated by pure model scaling or scratchpad-based reasoning. Our results reveal a fundamental mismatch between a model's learning behavior and desired skill compositions, with implications for reasoning reliability, out-of-distribution robustness, and alignment.

Shattered Compositionality: Counterintuitive Learning Dynamics of Transformers for Arithmetic

TL;DR

Transformers learning arithmetic subskills often deviate from human rule-based composition, exhibiting reverse and parallel acquisition that produces mixing errors, especially under distribution shifts. By training small transformers on synthetic tasks (addition, multiplication, comparison, sorting) and tracing subskill emergence with information-theoretic diagnostics, the authors show learning dynamics align with correlational signals in the data, not causal decompositions. These shattered compositionality behaviors persist across model scale, pretraining, and scratchpad reasoning, indicating a fundamental training-dynamics phenomenon with implications for reasoning reliability and robustness to out-of-distribution prompts. The work highlights the need for new evaluation and training strategies to mitigate brittle compositions in real-world LLM deployments and alignment contexts.

Abstract

Large language models (LLMs) often exhibit unexpected errors or unintended behavior, even at scale. While recent work reveals the discrepancy between LLMs and humans in skill compositions, the learning dynamics of skill compositions and the underlying cause of non-human behavior remain elusive. In this study, we investigate the mechanism of learning dynamics by training transformers on synthetic arithmetic tasks. Through extensive ablations and fine-grained diagnostic metrics, we discover that transformers do not reliably build skill compositions according to human-like sequential rules. Instead, they often acquire skills in reverse order or in parallel, which leads to unexpected mixing errors especially under distribution shifts--a phenomenon we refer to as shattered compositionality. To explain these behaviors, we provide evidence that correlational matching to the training data, rather than causal or procedural composition, shapes learning dynamics. We further show that shattered compositionality persists in modern LLMs and is not mitigated by pure model scaling or scratchpad-based reasoning. Our results reveal a fundamental mismatch between a model's learning behavior and desired skill compositions, with implications for reasoning reliability, out-of-distribution robustness, and alignment.
Paper Structure (79 sections, 1 theorem, 55 equations, 26 figures, 11 tables)

This paper contains 79 sections, 1 theorem, 55 equations, 26 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Static analyses of arithmetic and reasoning.
  4. Learning dynamics on arithmetic without decomposition.
  5. Skill composition and its limits in language models.
  6. Experimental Setup
  7. Tasks and data generation
  8. Addition task.
  9. Simple multiplication task.
  10. Comparison task.
  11. Sorting task.
  12. Model and Training
  13. Alternative setups.
  14. Counterintuitive Learning Dynamics of Arithmetic Subskills
  15. Addition
  16. ...and 64 more sections

Key Result

Theorem 5.1

Under uniform sampling, we have $I(a_1,e_0)>0$ and $I(a_i,e_j)=0$ for $j>0$. Moreover, $I(a_i,e_i|c_{i-1})>0$ for $i=1,2,3$.

Figures (26)

  • Figure 1: Transformers learn digits in non-human order for addition regardless of output formats. We train transformers on addition of the format $a+b+c+d=e$ with multiple-digit input integers $a,b,c,d$, and evaluate the digit-wise error rates of $e$ across 200K training steps. In contrast to human's natural order, models learn from higher digits to lower digits. This non-human order remains so even when we reverse the digit order of output $e$ which incentivizes the model to follow the human's order.
  • Figure 2: Transformers learn addition similar to an improving approximation algorithm, narrowing the error spread as training continues. When training the model for addition in reverse format, we compare the model's predicted integer $\widehat{e}$ and the groundtruth integer $e'$. The scatterplots are instances of the difference $e' - \widehat{e}$ collected over three intervals. We then fit a normal distribution for each scatterplot. Despite trained on raw digits, the model learns to approximate the correct solution with increasingly smaller Gaussian error.
  • Figure 3: Ablation experiments show randomizing higher digits disables learning. We train transformers with modified training data for addition with the reverse format, where the thousands-place digits are replaced by uniform digits in $\{0,1,2,3\}$. The model fails to learn any lower digit, whose digit-wise error rates are not better than random baseline.
  • Figure 4: Transformers learn digits in two-way order for simple multiplication. From the evolution of digit-wise error rates, we find two opposite sequential orders are learned simultaneously. (i) Reverse order: starting from the 41st (highest) digit to the 3rd digit; (ii) Normal order: starting from the 1st (lowest) digit.
  • Figure 5: Contrast pairs reveal competition between learning multiple skills for the comparison task. We use 4 test groups of contrast pairs (differing in single digits) to evaluate the model's ability to distinguish similar integers. The increase of errors at around 200 steps, particularly at thousands-place digits, suggests that learning "=" temporarily interfered learning the comparison of thousands-place digits.
  • ...and 21 more figures

Theorems & Definitions (2)

  • Theorem 5.1
  • proof