Can an Easy-to-Hard Curriculum Make Reasoning Emerge in Small Language Models? Evidence from a Four-Stage Curriculum on GPT-2

TL;DR

This work investigates whether a developmentally ordered, four‑stage curriculum can elicit transparent reasoning in small language models. Deploying Cognivolve on GPT‑2small (124M), the authors show that a staged progression from lexical to multi‑step symbolic reasoning yields an order‑of‑magnitude increase in specialized reasoning heads and redistributes them toward deeper layers, with higher attention entropy that balances local and global context. The curriculum also accelerates sample efficiency, achieving target performance in fewer optimization steps than a non‑curriculum baseline, though end‑of‑training final accuracy lags the baseline by about 32% unless further fine‑tuning is applied. These findings reveal a capacity‑accretion path for reasoning networks, improve interpretability via attention analyses, and point to design directions (mixed-stage fine‑tuning, richer probes) to reconcile transparency with ultimate task performance in small models.

Abstract

We demonstrate that a developmentally ordered curriculum markedly improves reasoning transparency and sample-efficiency in small language models (SLMs). Concretely, we train Cognivolve, a 124 M-parameter GPT-2 model, on a four-stage syllabus that ascends from lexical matching to multi-step symbolic inference and then evaluate it without any task-specific fine-tuning. Cognivolve reaches target accuracy in half the optimization steps of a single-phase baseline, activates an order-of-magnitude more gradient-salient reasoning heads, and shifts those heads toward deeper layers, yielding higher-entropy attention that balances local and long-range context. The same curriculum applied out of order or with optimizer resets fails to reproduce these gains, confirming that progression--not extra compute--drives the effect. We also identify open challenges: final-answer success still lags a conventional run by about 30%, and our saliency probe under-detects verbal-knowledge heads in the hardest stage, suggesting directions for mixed-stage fine-tuning and probe expansion.

Figures (8)

• Figure 1: Four-stage Cognivolve curriculum. Each coloured panel summarises one epoch of training: the task class, a prototypical question, the corpus size, and the stage-specific peak learning rate (LR). Difficulty ascends left → right—from one-step numeric or propositional queries to open-domain problems that require combining three or more facts and implicit world knowledge. We train the same GPT-2small weights continuously across stages; only the data partition and learning-rate ceiling change.
• Figure 2: Total number of specialized attention heads over training. Shaded regions denote one standard deviation across three seeds.
• Figure 3: Distribution of specialized heads across the 24 transformer layers at the final checkpoint.
• Figure 4: Validation success rate over training. Curriculum training ends after the final syllabus stage at $\sim$10 k steps; the baseline continues to 60 k.
• Figure 5: Step-by-step reasoning accuracy over the same training runs.