Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns

Haoyue Bai, Yiyou Sun, Wenjie Hu, Shi Qiu, Maggie Ziyu Huan, Peiyang Song, Robert Nowak, Dawn Song

TL;DR

The paper tackles why supervised fine-tuning and reinforcement-learning tuning yield different generalization in large language models by decomposing reasoning into five atomic cognitive skills: calculation, enumeration, simulation, fact retrieval, and diagnostic checking. It introduces a three-stage benchmark and a behavior–domain grid to measure these skills across math, science, coding, and non-reasoning domains, linking behavioral profiles to low-level statistical patterns. Across experiments with Qwen-derived models, RL consistently yields balanced skill distributions and better cross-domain transfer, while SFT induces over-specialization and surface-pattern reliance; long CoT can partially mitigate cross-domain gaps. The findings advocate for training strategies that reinforce core skills and monitor representations, guiding future design of robust, interpretable, and transferable reasoning in LLMs.

Abstract

Large Language Models (LLMs) display strikingly different generalization behaviors: supervised fine-tuning (SFT) often narrows capability, whereas reinforcement-learning (RL) tuning tends to preserve it. The reasons behind this divergence remain unclear, as prior studies have largely relied on coarse accuracy metrics. We address this gap by introducing a novel benchmark that decomposes reasoning into atomic core skills such as calculation, fact retrieval, simulation, enumeration, and diagnostic, providing a concrete framework for addressing the fundamental question of what constitutes reasoning in LLMs. By isolating and measuring these core skills, the benchmark offers a more granular view of how specific cognitive abilities emerge, transfer, and sometimes collapse during post-training. Combined with analyses of low-level statistical patterns such as distributional divergence and parameter statistics, it enables a fine-grained study of how generalization evolves under SFT and RL across mathematical, scientific reasoning, and non-reasoning tasks. Our meta-probing framework tracks model behavior at different training stages and reveals that RL-tuned models maintain more stable behavioral profiles and resist collapse in reasoning skills, whereas SFT models exhibit sharper drift and overfit to surface patterns. This work provides new insights into the nature of reasoning in LLMs and points toward principles for designing training strategies that foster broad, robust generalization.

How and Why LLMs Generalize: A Fine-Grained Analysis of LLM Reasoning from Cognitive Behaviors to Low-Level Patterns

TL;DR

The paper tackles why supervised fine-tuning and reinforcement-learning tuning yield different generalization in large language models by decomposing reasoning into five atomic cognitive skills: calculation, enumeration, simulation, fact retrieval, and diagnostic checking. It introduces a three-stage benchmark and a behavior–domain grid to measure these skills across math, science, coding, and non-reasoning domains, linking behavioral profiles to low-level statistical patterns. Across experiments with Qwen-derived models, RL consistently yields balanced skill distributions and better cross-domain transfer, while SFT induces over-specialization and surface-pattern reliance; long CoT can partially mitigate cross-domain gaps. The findings advocate for training strategies that reinforce core skills and monitor representations, guiding future design of robust, interpretable, and transferable reasoning in LLMs.

Abstract

Large Language Models (LLMs) display strikingly different generalization behaviors: supervised fine-tuning (SFT) often narrows capability, whereas reinforcement-learning (RL) tuning tends to preserve it. The reasons behind this divergence remain unclear, as prior studies have largely relied on coarse accuracy metrics. We address this gap by introducing a novel benchmark that decomposes reasoning into atomic core skills such as calculation, fact retrieval, simulation, enumeration, and diagnostic, providing a concrete framework for addressing the fundamental question of what constitutes reasoning in LLMs. By isolating and measuring these core skills, the benchmark offers a more granular view of how specific cognitive abilities emerge, transfer, and sometimes collapse during post-training. Combined with analyses of low-level statistical patterns such as distributional divergence and parameter statistics, it enables a fine-grained study of how generalization evolves under SFT and RL across mathematical, scientific reasoning, and non-reasoning tasks. Our meta-probing framework tracks model behavior at different training stages and reveals that RL-tuned models maintain more stable behavioral profiles and resist collapse in reasoning skills, whereas SFT models exhibit sharper drift and overfit to surface patterns. This work provides new insights into the nature of reasoning in LLMs and points toward principles for designing training strategies that foster broad, robust generalization.
Paper Structure (44 sections, 5 equations, 12 figures, 3 tables)

This paper contains 44 sections, 5 equations, 12 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Illustrative Example.
  3. Related Works
  4. Decompose “Reasoning” into Fine-grained Cognitive Behaviors
  5. Benchmark Construction
  6. Behavior–Domain Grid
  7. Experiments
  8. Cognitive-skill Profiles
  9. Cross-Domain Generalization
  10. Linking Behaviors to Sparse Latent Features
  11. Low-Level Statistical Shifts and Surface Patterns
  12. Parameter-space shifts under post-training.
  13. Component-level differences.
  14. Summary and implications for future study
  15. Additional Experimental Details
  16. ...and 29 more sections

Figures (12)

  • Figure 1: Decomposing reasoning into atomic cognitive skills. A spring–block example is solved step by step: fact retrieval, simulation, calculation, and diagnostic, checking each function as an isolated cognitive skill. When combined, these skills form a coherent reasoning trace that enables solving real-world problems, illustrating how complex reasoning emerges from the composition of simpler, specialized components.
  • Figure 2: Overview of our three-stage benchmark construction pipeline. (1) Seed Design defines atomic seeds by domain and behavior; (2) Candidate Retrieval expands them via embedding search over public datasets; (3) Manual Verification filters for coverage, diversity, and difficulty, yielding a curated benchmark for fine-grained reasoning analysis.
  • Figure 3: Radar plots of five cognitive behaviors (calculation, enumeration, simulation, fact retrieval, diagnostic checking). Each panel contrasts SFT (blue) vs. RL (orange) for (a) Math-14B, (b) Physics-14B, (c) Math-4B, and (d) Physics-4B. RL is more balanced; SFT often concentrates on a few skills.
  • Figure 4: Radar plots of post-training effects on math (a) and non-reasoning (b) using math-trained models. Both transfer weakly to non-reasoning; RL preserves a more balanced skill profile, whereas SFT shows uneven gains/losses out of domain.
  • Figure 5: Shape comparison of SFT profiles with thinking (orange) vs no thinking (blue) on (a) Math and (b) Physics. Compared to the think model, the no-think model is even more spiky and calculation-dominated. All models are derived from the Qwen3-14B base; SFT and RL share the same base.
  • ...and 7 more figures