CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models

TL;DR

CurLL presents a curriculum-grounded framework for evaluating continual learning in language models by aligning skill acquisition with human developmental stages. It introduces a large-scale synthetic dataset (23.4B tokens) and a detailed 1,300+ indicator skill graph to enable fine-grained analysis of forgetting, forward transfer, and backward transfer across staged curricula. Experiments with independent, joint, and sequential training on a 135M-parameter transformer demonstrate that the order of data and the underlying skill dependencies shape learning dynamics beyond traditional benchmarks. The work offers a scalable, interpretable evaluation platform and outlines avenues to extend developmental curricula to broader age ranges and larger models.

Abstract

We introduce a comprehensive continual learning dataset and benchmark (CurlL) grounded in human developmental trajectories from ages 5-10, enabling systematic and fine-grained assessment of models' ability to progressively acquire new skills. CurlL spans five developmental stages (0-4) covering ages 5-10, supported by a skill graph that breaks down broad skills into smaller abilities, concrete goals, and measurable indicators, while also capturing which abilities build on others. We generate a 23.4B-token synthetic dataset with controlled skill progression, vocabulary complexity, and format diversity, comprising paragraphs, comprehension-based QA (CQA), skill-testing QA (CSQA), and instruction-response (IR) pairs. Stage-wise token counts range from 2.12B to 6.78B tokens, supporting precise analysis of forgetting, forward transfer, and backward transfer. Using a 135M-parameter transformer trained under independent, joint, and sequential (continual) setups, we show trade-offs in skill retention and transfer efficiency. By mirroring human learning patterns and providing fine-grained control over skill dependencies, this work advances continual learning evaluations for language models.

Figures (8)

• Figure 1: Developmental framework for children aged 0-11 years, categorized into stages (0-4). Only examples of skills and subskills are mentioned here. An example of how the data looks like is given in the top right. Two nodes and an edge from the skill graph is given in the bottom right.
• Figure 2: Heatmap showing the number of prerequisite edges in the skill graph. Rows correspond to source stages, columns to target stages, and color intensity indicates the number of connections.
• Figure 3: Synthetic data generation pipeline
• Figure 4: Stage-wise evaluation results for different training setups. Independent training corresponds to models trained on a single stage, joint training to models trained on mixtures of data up to a stage, and continual training to sequential upto a stage. Heatmaps report summed correctness scores across all test formats (IR, CQA, CSQA)
• Figure 5: Forgetting analysis across training setups. The plots show performance differences between joint and continual training for IR, CQA, and CSQA test sets across stages 0-4. The Y-axis corresponds to models trained upto a stage. The X-axis corresponds to test set of mentioned stage.