Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

Atsuki Yamaguchi, Maggie Mi, Nikolaos Aletras

TL;DR

The paper addresses the limited linguistic competence of decoder-based language models trained with standard next-token objectives by introducing L2T, a pre-training framework that interleaves 14 language-learning tasks with raw text to produce structured input-output signals. Through experiments on 500M and 1B parameter models under Disjoint and Shared data regimes, L2T yields substantial improvements in linguistic competence on BLiMP (up to 11.3 points, average around 2.8) while largely preserving general reasoning abilities, and it accelerates the acquisition of linguistic knowledge. Analysis shows the strongest gains arise in long-distance syntactic dependencies and island constraints, with some phenomena saturating or remaining challenging, indicating the need for targeted discourse-level objectives. The findings support a data-centric view where structured linguistic supervision during pre-training complements world-knowledge learning, and they highlight important trade-offs when balancing raw text and linguistic tasks, guiding future multilingual extensions and curriculum-based scaling.

Abstract

Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction. Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation. Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates its acquisition, while maintaining competitive performance on general reasoning tasks.

Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks

TL;DR

The paper addresses the limited linguistic competence of decoder-based language models trained with standard next-token objectives by introducing L2T, a pre-training framework that interleaves 14 language-learning tasks with raw text to produce structured input-output signals. Through experiments on 500M and 1B parameter models under Disjoint and Shared data regimes, L2T yields substantial improvements in linguistic competence on BLiMP (up to 11.3 points, average around 2.8) while largely preserving general reasoning abilities, and it accelerates the acquisition of linguistic knowledge. Analysis shows the strongest gains arise in long-distance syntactic dependencies and island constraints, with some phenomena saturating or remaining challenging, indicating the need for targeted discourse-level objectives. The findings support a data-centric view where structured linguistic supervision during pre-training complements world-knowledge learning, and they highlight important trade-offs when balancing raw text and linguistic tasks, guiding future multilingual extensions and curriculum-based scaling.

Abstract

Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. While this approach facilitates the learning of world knowledge and reasoning, it does not explicitly optimize for linguistic competence. To bridge this gap, we propose L2T, a pre-training framework integrating Language Learning Tasks alongside standard next-token prediction. Inspired by human language acquisition, L2T transforms raw text into structured input-output pairs to provide explicit linguistic stimulation. Pre-training LMs on a mixture of raw text and L2T data not only improves overall performance on linguistic competence benchmarks but accelerates its acquisition, while maintaining competitive performance on general reasoning tasks.
Paper Structure (58 sections, 8 figures, 6 tables)

This paper contains 58 sections, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. L2T: Language Learning Tasks
  3. Experimental Setup
  4. Pre-training Data Scenarios
  5. Disjoint (Abundant Data ).
  6. Shared (Limited Data ).
  7. Training Configuration
  8. Evaluation Framework
  9. Baselines.
  10. Linguistic Competence.
  11. General Benchmarks.
  12. Results
  13. Linguistic Competence
  14. General Benchmarks
  15. Disjoint.
  16. ...and 43 more sections

Figures (8)

  • Figure 1: L2T vs. standard CLM over raw text.
  • Figure 2: Accuracy by linguistic subfield in BLiMP between Raw and L2T across model sizes and training steps using Disjoint Raw and L2T data.
  • Figure 3: Overview of the 14 language learning tasks. Colors denote linguistic granularity: character (blue), word (green), sentence (orange), and discourse (purple).
  • Figure 4: Linguistic competence comparisons on BLiMP between different L2T models trained on specific 25B token single task data.
  • Figure 5: Linguistic competence comparisons on BLiMP between different L2T models trained on specific 25B token single task data.
  • ...and 3 more figures