Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KidLM: Advancing Language Models for Children -- Early Insights and Future Directions

Mir Tafseer Nayeem, Davood Rafiei

TL;DR

This work introduces KidLM, a child-focused language-model effort grounded in a user-centric pre-training corpus and a novel Stratified Masking objective. By curating high-quality, kid-written content and validating it with editors, the authors aim to instill safety, lexical simplicity, and responsiveness to children’s needs. Empirical results show improved understanding of lower-grade text and reduced risk of generating stereotypes or toxic content, with KidLM+ further enhancing safety and preference alignment through stratified masking. The paper also outlines a data-centric research agenda, emphasizing human-centered evaluation, broader harm coverage, and environmentally mindful training practices. Overall, it demonstrates that high-quality, audience-aware pre-training data is essential for effective and safe child-oriented language modeling.

Abstract

Recent studies highlight the potential of large language models in creating educational tools for children, yet significant challenges remain in maintaining key child-specific properties such as linguistic nuances, cognitive needs, and safety standards. In this paper, we explore foundational steps toward the development of child-specific language models, emphasizing the necessity of high-quality pre-training data. We introduce a novel user-centric data collection pipeline that involves gathering and validating a corpus specifically written for and sometimes by children. Additionally, we propose a new training objective, Stratified Masking, which dynamically adjusts masking probabilities based on our domain-specific child language data, enabling models to prioritize vocabulary and concepts more suitable for children. Experimental evaluations demonstrate that our model excels in understanding lower grade-level text, maintains safety by avoiding stereotypes, and captures children's unique preferences. Furthermore, we provide actionable insights for future research and development in child-specific language modeling.

KidLM: Advancing Language Models for Children -- Early Insights and Future Directions

TL;DR

This work introduces KidLM, a child-focused language-model effort grounded in a user-centric pre-training corpus and a novel Stratified Masking objective. By curating high-quality, kid-written content and validating it with editors, the authors aim to instill safety, lexical simplicity, and responsiveness to children’s needs. Empirical results show improved understanding of lower-grade text and reduced risk of generating stereotypes or toxic content, with KidLM+ further enhancing safety and preference alignment through stratified masking. The paper also outlines a data-centric research agenda, emphasizing human-centered evaluation, broader harm coverage, and environmentally mindful training practices. Overall, it demonstrates that high-quality, audience-aware pre-training data is essential for effective and safe child-oriented language modeling.

Abstract

Recent studies highlight the potential of large language models in creating educational tools for children, yet significant challenges remain in maintaining key child-specific properties such as linguistic nuances, cognitive needs, and safety standards. In this paper, we explore foundational steps toward the development of child-specific language models, emphasizing the necessity of high-quality pre-training data. We introduce a novel user-centric data collection pipeline that involves gathering and validating a corpus specifically written for and sometimes by children. Additionally, we propose a new training objective, Stratified Masking, which dynamically adjusts masking probabilities based on our domain-specific child language data, enabling models to prioritize vocabulary and concepts more suitable for children. Experimental evaluations demonstrate that our model excels in understanding lower grade-level text, maintains safety by avoiding stereotypes, and captures children's unique preferences. Furthermore, we provide actionable insights for future research and development in child-specific language modeling.
Paper Structure (48 sections, 4 equations, 3 figures, 17 tables)

This paper contains 48 sections, 4 equations, 3 figures, 17 tables.

Table of Contents

  1. Introduction
  2. KidLM Construction
  3. KidLM Corpus
  4. User-Centric
  5. Source Identification
  6. Manual Data Verification
  7. Quality Filtering
  8. Additional Filtering
  9. Data Diversity
  10. Data Quantity
  11. KidLM Models
  12. Stratified Masking
  13. Stopwords
  14. Dale-Chall Easy Words List
  15. Other Words
  16. ...and 33 more sections

Figures (3)

  • Figure 1: User-Centric Data Collection Pipeline for our KidLM (corpus).
  • Figure 2: Venn diagram illustrating different word classes used in our proposed Stratified Masking.
  • Figure 3: (a) In default random masking, all words have a equal probability of 0.15 of being masked. (b) In our proposed stratified masking, stopwords are masked with a probability of 0.15, Dale-Chall words with a probability of 0.20, and other words with a probability of 0.25, to enhance learning focus on kid-specific words.