Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating LLMs on Generating Age-Appropriate Child-Like Conversations

Syed Zohaib Hassan, Pål Halvorsen, Miriam S. Johnson, Pierre Lison

TL;DR

This study benchmarks five LLMs on generating age-appropriate, child-like Norwegian dialogue for 5- and 9-year-olds using a blind, educator-led evaluation against real child interviews. It combines human judgments with computational linguistic analysis and a Norwegian child-lexicon resource to assess developmental authenticity. Results show a systematic bias toward linguistically mature language across models, with higher accuracy for younger children and strong inter-rater reliability among evaluators. The findings underscore data scarcity and adult-centric training as key bottlenecks, while offering a practical evaluation framework and directions for building domain-specific resources and prompt strategies in low-resource languages.

Abstract

Large Language Models (LLMs), predominantly trained on adult conversational data, face significant challenges when generating authentic, child-like dialogue for specialized applications. We present a comparative study evaluating five different LLMs (GPT-4, RUTER-LLAMA-2-13b, GPTSW, NorMistral-7b, and NorBloom-7b) to generate age-appropriate Norwegian conversations for children aged 5 and 9 years. Through a blind evaluation by eleven education professionals using both real child interview data and LLM-generated text samples, we assessed authenticity and developmental appropriateness. Our results show that evaluators achieved strong inter-rater reliability (ICC=0.75) and demonstrated higher accuracy in age prediction for younger children (5-year-olds) compared to older children (9-year-olds). While GPT-4 and NorBloom-7b performed relatively well, most models generated language perceived as more linguistically advanced than the target age groups. These findings highlight critical data-related challenges in developing LLM systems for specialized applications involving children, particularly in low-resource languages where comprehensive age-appropriate lexical resources are scarce.

Evaluating LLMs on Generating Age-Appropriate Child-Like Conversations

TL;DR

This study benchmarks five LLMs on generating age-appropriate, child-like Norwegian dialogue for 5- and 9-year-olds using a blind, educator-led evaluation against real child interviews. It combines human judgments with computational linguistic analysis and a Norwegian child-lexicon resource to assess developmental authenticity. Results show a systematic bias toward linguistically mature language across models, with higher accuracy for younger children and strong inter-rater reliability among evaluators. The findings underscore data scarcity and adult-centric training as key bottlenecks, while offering a practical evaluation framework and directions for building domain-specific resources and prompt strategies in low-resource languages.

Abstract

Large Language Models (LLMs), predominantly trained on adult conversational data, face significant challenges when generating authentic, child-like dialogue for specialized applications. We present a comparative study evaluating five different LLMs (GPT-4, RUTER-LLAMA-2-13b, GPTSW, NorMistral-7b, and NorBloom-7b) to generate age-appropriate Norwegian conversations for children aged 5 and 9 years. Through a blind evaluation by eleven education professionals using both real child interview data and LLM-generated text samples, we assessed authenticity and developmental appropriateness. Our results show that evaluators achieved strong inter-rater reliability (ICC=0.75) and demonstrated higher accuracy in age prediction for younger children (5-year-olds) compared to older children (9-year-olds). While GPT-4 and NorBloom-7b performed relatively well, most models generated language perceived as more linguistically advanced than the target age groups. These findings highlight critical data-related challenges in developing LLM systems for specialized applications involving children, particularly in low-resource languages where comprehensive age-appropriate lexical resources are scarce.

Paper Structure

This paper contains 25 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Model Selection and Description
  5. Real Child Interview Data Collection
  6. LLM Text Generation Protocol
  7. Norwegian Linguistic Resources
  8. Evaluation Methodology
  9. Evaluation Protocol
  10. Computational Linguistic Analysis
  11. Results
  12. Education Professional Age Prediction Performance
  13. Age Prediction Accuracy and Model Comparison
  14. Age Group Differences in Prediction Accuracy
  15. Inter-Rater Reliability Analysis
  16. ...and 10 more sections

Figures (3)

  • Figure 1: Age prediction statistics by age for consolidated dataset
  • Figure 2: Age prediction histograms for age 5
  • Figure 3: Age prediction histograms for age 9