Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)

Francesca Padovani, Bastian Bunzeck, Manar Ali, Omar Momen, Arianna Bisazza, Hendrik Buschmeier, Sina Zarrieß

TL;DR

This study probes whether pretraining a small BabyLM on child–caregiver dialogue alone can yield linguistically proficient models. It introduces llamalogue, a 135M-parameter model trained on CHILDES dialogue triplets, and applies two reinforcement-learning paradigms—DPO and PPO—to promote more communicative dialogue generation. Across standard BabyLM benchmarks, the base model and PPO/DPO-finetuned variants largely underperform compared with strong baselines, though DPO improves performance on a custom dialogue minimal-pair benchmark. The results suggest CDS-only pretraining yields domain-specific gains (dialogue coherence) but limited transfer to broader linguistic capabilities, highlighting the need for curriculum design and richer data mixtures in developmentally inspired language learning.

Abstract

We investigate whether pre-training exclusively on dialogue data results in formally and functionally apt small language models. Based on this pre-trained llamalogue model, we employ a variety of fine-tuning strategies to enforce "more communicative" text generations by our models. Although our models underperform on most standard BabyLM benchmarks, they excel at dialogue continuation prediction in a minimal pair setting. While PPO fine-tuning has mixed to adversarial effects on our models, DPO fine-tuning further improves their performance on our custom dialogue benchmark.

Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)

TL;DR

This study probes whether pretraining a small BabyLM on child–caregiver dialogue alone can yield linguistically proficient models. It introduces llamalogue, a 135M-parameter model trained on CHILDES dialogue triplets, and applies two reinforcement-learning paradigms—DPO and PPO—to promote more communicative dialogue generation. Across standard BabyLM benchmarks, the base model and PPO/DPO-finetuned variants largely underperform compared with strong baselines, though DPO improves performance on a custom dialogue minimal-pair benchmark. The results suggest CDS-only pretraining yields domain-specific gains (dialogue coherence) but limited transfer to broader linguistic capabilities, highlighting the need for curriculum design and richer data mixtures in developmentally inspired language learning.

Abstract

We investigate whether pre-training exclusively on dialogue data results in formally and functionally apt small language models. Based on this pre-trained llamalogue model, we employ a variety of fine-tuning strategies to enforce "more communicative" text generations by our models. Although our models underperform on most standard BabyLM benchmarks, they excel at dialogue continuation prediction in a minimal pair setting. While PPO fine-tuning has mixed to adversarial effects on our models, DPO fine-tuning further improves their performance on our custom dialogue benchmark.
Paper Structure (35 sections, 3 equations, 3 figures, 7 tables)

This paper contains 35 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Learning exclusively from CDS
  4. Cognitively/developmentally plausible RL
  5. Methodology
  6. Pretraining
  7. Data
  8. Base model
  9. DPO fine-tuning
  10. Naturalistic data
  11. Synthetic data
  12. PPO fine-tuning
  13. 1-gram BLEU Reward
  14. Semantic Similarity Reward
  15. LLM-generated Reward
  16. ...and 20 more sections

Figures (3)

  • Figure 1: Training loss (left) and reward trends (appropriate vs. random) during training (right) for both DPO models.
  • Figure 2: Reward trends over training steps for four reward metrics: 1-gram BLEU, Semantic similarity, LLM score, and Teacher Confidence-based. Vertical line marks the end of epoch 1. For LLM score and confidence, the $y$-axis range has been restricted to enhance visibility of trends, and does not represent the full possible reward scale.
  • Figure 3: Learning trajectories for our base model across pre-training for 10 epochs. Note that the $x$-axis is log-scaled to make the very early training dynamics more visible.