Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models

Raha Askari, Sina Zarrieß, Özge Alacam, Judith Sieker

TL;DR

This paper develops a pragmatic evaluation framework for BabyLMs by adapting the Conversational Violations Test (CVT) to a large, child-directed dataset of 2,250 dialogues spanning five Gricean maxims. It benchmarks eight BabyLM baselines across two data tracks (<10M and <100M tokens) and compares them to a large, three-trillion-token model (OLMo-1B) and to child performance. Results show modest gains with increased data, yet BabyLMs remain below child-level and large-model competence, particularly on informativeness-related maxims, while large-scale pretraining helps with information structure but not social pragmatics. The study argues for dedicated, linguistically grounded pragmatic benchmarks to track development and guide data-efficient model design, and provides a scalable resource for cross-model pragmatics evaluation, including future multilingual extensions and architecture diversification.

Abstract

Implicit meanings are integral to human communication, making it essential for language models to be capable of identifying and interpreting them. Grice (1975) proposed a set of conversational maxims that guide cooperative dialogue, noting that speakers may deliberately violate these principles to express meanings beyond literal words, and that listeners, in turn, recognize such violations to draw pragmatic inferences. Building on Surian et al. (1996)'s study of children's sensitivity to violations of Gricean maxims, we introduce a novel benchmark to test whether language models pretrained on less than 10M and less than 100M tokens can distinguish maxim-adhering from maxim-violating utterances. We compare these BabyLMs across five maxims and situate their performance relative to children and a Large Language Model (LLM) pretrained on 3T tokens. We find that overall, models trained on less than 100M tokens outperform those trained on less than 10M, yet fall short of child-level and LLM competence. Our results suggest that modest data increases improve some aspects of pragmatic behavior, leading to finer-grained differentiation between pragmatic dimensions.

Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models

TL;DR

This paper develops a pragmatic evaluation framework for BabyLMs by adapting the Conversational Violations Test (CVT) to a large, child-directed dataset of 2,250 dialogues spanning five Gricean maxims. It benchmarks eight BabyLM baselines across two data tracks (<10M and <100M tokens) and compares them to a large, three-trillion-token model (OLMo-1B) and to child performance. Results show modest gains with increased data, yet BabyLMs remain below child-level and large-model competence, particularly on informativeness-related maxims, while large-scale pretraining helps with information structure but not social pragmatics. The study argues for dedicated, linguistically grounded pragmatic benchmarks to track development and guide data-efficient model design, and provides a scalable resource for cross-model pragmatics evaluation, including future multilingual extensions and architecture diversification.

Abstract

Implicit meanings are integral to human communication, making it essential for language models to be capable of identifying and interpreting them. Grice (1975) proposed a set of conversational maxims that guide cooperative dialogue, noting that speakers may deliberately violate these principles to express meanings beyond literal words, and that listeners, in turn, recognize such violations to draw pragmatic inferences. Building on Surian et al. (1996)'s study of children's sensitivity to violations of Gricean maxims, we introduce a novel benchmark to test whether language models pretrained on less than 10M and less than 100M tokens can distinguish maxim-adhering from maxim-violating utterances. We compare these BabyLMs across five maxims and situate their performance relative to children and a Large Language Model (LLM) pretrained on 3T tokens. We find that overall, models trained on less than 100M tokens outperform those trained on less than 10M, yet fall short of child-level and LLM competence. Our results suggest that modest data increases improve some aspects of pragmatic behavior, leading to finer-grained differentiation between pragmatic dimensions.

Paper Structure

This paper contains 20 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Developmental Studies.
  4. BabyLMs.
  5. Pragmatic Evaluation in LMs.
  6. Approach
  7. Data.
  8. Models.
  9. Evaluation.
  10. Results
  11. Results by Gricean Maxim.
  12. Inter-maxim Correlations.
  13. Effects of Architecture and Model size.
  14. Comparison to Child Performance.
  15. Summary and Discussion.
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Pearson correlations between per-model accuracies across maxims. Top: Strict-small track. Bottom: Strict track.