DevBench: A multimodal developmental benchmark for language learning

TL;DR

DevBench addresses how vision-language models align with children's language development by providing seven developmentally informed, multimodal tasks with child and adult human data. It introduces a response-pattern comparison framework using softmax-optimised KL divergence and RSA to quantify human-likeness across lexical, syntactic, and semantic levels, and analyzes model learning trajectories through intermediate OpenCLIP checkpoints. Key findings show that higher task accuracy and larger models tend to be more human-like, while semantic representations can diverge with training, and that developmentally realistic data yield nuanced trajectory patterns. The benchmark offers a principled tool for diagnosing where models diverge from human learning and informs future directions for data-efficient, human-aligned language models.

Abstract

How (dis)similar are the learning trajectories of vision-language models and children? Recent modeling work has attempted to understand the gap between models' and humans' data efficiency by constructing models trained on less data, especially multimodal naturalistic data. However, such models are often evaluated on adult-level benchmarks, with limited breadth in language abilities tested, and without direct comparison to behavioral data. We introduce DevBench, a multimodal benchmark comprising seven language evaluation tasks spanning the domains of lexical, syntactic, and semantic ability, with behavioral data from both children and adults. We evaluate a set of vision-language models on these tasks, comparing models and humans not only on accuracy but on their response patterns. Across tasks, models exhibit variation in their closeness to human response patterns, and models that perform better on a task also more closely resemble human behavioral responses. We also examine the developmental trajectory of OpenCLIP over training, finding that greater training results in closer approximations to adult response patterns. DevBench thus provides a benchmark for comparing models to human language development. These comparisons highlight ways in which model and human language learning processes diverge, providing insight into entry points for improving language models.

Figures (5)

• Figure 1: Tasks in DevBench arranged by linguistic domain, along with the ages for which corresponding human data are available. A: Adult.
• Figure 2: Sample trials for each task in DevBench.
• Figure 3: (a) Correlations between model--human similarity and task accuracy, log number of parameters (size), and log number of training images for each task (training), averaged across ages. Accuracy correlates the most strongly with model--human similarity, followed by size, then training. (b) Model--human dissimilarity as a function of task accuracy for each model on the Visual Vocabulary task. Higher performing models showed a closer correspondence to behavioral patterns from children and adults. A: Adult.
• Figure 4: Trajectories of model--human similarity for VV, TROG, and WG. OpenCLIP-H becomes more human-like over training, and recovers developmental trends for VV.
• Figure 5: Trajectories of model--human similarity for LWL, WAT, VOC, and THINGS. Note that the three age groups for LWL are not comparable because they use different stimuli sets. For LWL and WAT, smaller values indicate greater model--human similarity, whereas for VOC and THINGS, larger values indicate greater model--human similarity.