End-to-End Joint ASR and Speaker Role Diarization with Child-Adult Interactions

TL;DR

This work tackles the challenge of generating speaker-attributed transcripts for child–adult interactions by proposing a single end-to-end model that jointly performs ASR and child–adult speaker-role diarization. It extends the Whisper encoder–decoder with a serialized output training scheme, a frame-level diarization head, diarization-guided silence suppression, and a state-machine enforced decoding mechanism to produce structurally valid outputs with accurate temporal boundaries. Empirical results on Playlogue and ADOS-Mod3 show consistent mtWER improvements over cascaded baselines and competitive DER, demonstrating that end-to-end joint modeling reduces error propagation and yields coherent, speaker-labeled transcripts at scale. The approach enables automatic extraction of behavioral and conversational metrics, supporting scalable clinical research in child development and ASD. The work also provides ablations and analyses validating the roles of timestamp and diarization supervision in shaping encoder representations and decoding reliability, underscoring the practical value of integrating ASR, diarization, and timing in a unified framework.

Abstract

Accurate transcription and speaker diarization of child-adult spoken interactions are crucial for developmental and clinical research. However, manual annotation is time-consuming and challenging to scale. Existing automated systems typically rely on cascaded speaker diarization and speech recognition pipelines, which can lead to error propagation. This paper presents a unified end-to-end framework that extends the Whisper encoder-decoder architecture to jointly model ASR and child-adult speaker role diarization. The proposed approach integrates: (i) a serialized output training scheme that emits speaker tags and start/end timestamps, (ii) a lightweight frame-level diarization head that enhances speaker-discriminative encoder representations, (iii) diarization-guided silence suppression for improved temporal precision, and (iv) a state-machine-based forced decoding procedure that guarantees structurally valid outputs. Comprehensive evaluations on two datasets demonstrate consistent and substantial improvements over two cascaded baselines, achieving lower multi-talker word error rates and demonstrating competitive diarization accuracy across both Whisper-small and Whisper-large models. These findings highlight the effectiveness and practical utility of the proposed joint modeling framework for generating reliable, speaker-attributed transcripts of child-adult interactions at scale. The code and model weights are publicly available

Figures (5)

• Figure 1: Baseline speaker diarization pipeline with Whisper Encoder.
• Figure 2: Proposed joint ASR and speaker diarization training architecture. The last hidden layer from the encoder is used for the decoder and diarization head.
• Figure 3: State-machine diagram for forced decoding during inference.
• Figure 4: Error components for mtWER. The yellow and blue colors highlight two separate speaker roles. REF is the ground-truth transcript, while HYP is the inferred hypothesis.
• Figure 5: Child vs adult kNN classification on utterance-level encoder outputs. T and DH denote Timestamp and Diarization Head, respectively.