Can large audio language models understand child stuttering speech? speech summarization, and source separation

TL;DR

Can large audio-language models reliably understand child stuttering speech and generate child-only summaries from mixed audio? The paper evaluates multiple LALMs on two settings (interview and reading) with tasks for single-channel source separation and child-focused summarization, using a combination of automatic metrics, human expert ratings, and LLM-based judgments. It introduces speaker-purity constraints and two reproducible pipelines (text-first and audio-first), and assesses models on the FluencyBank English Voices dataset with 22 child participants. Findings show that certain audio-first models (notably Audio Flamingo-3) can produce clinically meaningful, child-centered summaries, but leakage and disfluency preservation remain challenging, and standard lexical metrics may not align with human judgments. The work offers prompts, evaluation scripts, and practical guidance for deploying pediatric LALM tools in education and clinical contexts, along with avenues for future improvements in diarization, prompting, and cross-accent generalization.

Abstract

Child speech differs from adult speech in acoustics, prosody, and language development, and disfluencies (repetitions, prolongations, blocks) further challenge Automatic Speech Recognition (ASR) and downstream Natural Language Processing (NLP). Recent large audio-language models (LALMs) demonstrate strong cross-modal audio understanding; however, their behavior in disfluent child speech remains underexplored. We evaluate several state-of-the-art LALMs in two settings: an interview (mixed speakers) and a reading task (single child). The tasks are (i) single-channel source separation to isolate the child and (ii) child-only summarization that preserves clinically relevant disfluencies and avoids adult-speech leakage. Evaluation combines Large Language Model (LLM) as a judge, human expert ratings, and BERTScore (F1), and we report agreement between models and between models and humans to assess reliability. Our findings delineate the conditions under which LALMs produce faithful child-only summaries from mixed audio and where they fail, offering practical guidance for clinical and educational deployments. We provide prompts and evaluation scripts to support replication.

Figures (1)

• Figure 1: Overview of the evaluation, including the LALMs used for summarization and LLMs and Human expert evaluations.