Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization

Máté Gedeon, Péter Mihajlik

TL;DR

LibriConvo addresses the scarcity of realistic multi-speaker conversational data for diarization and ASR by generating synthetic dialogues with semantic coherence and natural timing. The pipeline combines Speaker-Aware Simulated Conversation (SASC) with LibriTTS utterances, CallHome boundary signals, temporal compression, and a physics-informed RIR selection to produce acoustically plausible dialogues. It yields 240.1 hours across 1,496 dialogues and 830 speakers with speaker-disjoint splits, enabling robust benchmarking; baselines show Sortformer outperforming pyannote for diarization, and a fine-tuned fastconformer-ctc-xl with Serialized Output Training achieving leading cpWER (e.g., 6.97% cpWER on the test set). The dataset is publicly available on Hugging Face, offering a reproducible, scalable resource for advancing multi-speaker speech processing in controlled yet realistic conversational settings.

Abstract

We introduce LibriConvo, a simulated multi-speaker conversational dataset based on speaker-aware conversation simulation (SASC), designed to support training and evaluation of speaker diarization and automatic speech recognition (ASR) systems. Unlike prior resources that mostly rely on semantically disconnected utterances and implausible temporal gaps, LibriConvo ensures semantic coherence and realistic conversational timing. Our pipeline leverages CallHome with external VAD for reliable boundaries, applies compression to reduce unnaturally long silences, and organizes LibriTTS utterances by book to maintain contextual consistency. Acoustic realism is enhanced via a novel room impulse response selection procedure that ranks speaker-microphone configurations by spatial plausibility, balancing realism and diversity. The dataset comprises 240.1 hours across 1,496 dialogues with 830 unique speakers, split in a speaker-disjoint manner for robust evaluation. Baselines show that the sortformer model outperforms the pyannote pipeline in diarization, while a fine-tuned Fast Conformer-CTC XLarge with Serialized Output Training achieves 7.29\% WER for ASR, surpassing zero-shot Whisper-large-v3. LibriConvo provides a valuable resource for advancing multi-speaker speech processing research with realistic conversational dynamics and controlled experimental conditions.

LibriConvo: Simulating Conversations from Read Literature for ASR and Diarization

TL;DR

LibriConvo addresses the scarcity of realistic multi-speaker conversational data for diarization and ASR by generating synthetic dialogues with semantic coherence and natural timing. The pipeline combines Speaker-Aware Simulated Conversation (SASC) with LibriTTS utterances, CallHome boundary signals, temporal compression, and a physics-informed RIR selection to produce acoustically plausible dialogues. It yields 240.1 hours across 1,496 dialogues and 830 speakers with speaker-disjoint splits, enabling robust benchmarking; baselines show Sortformer outperforming pyannote for diarization, and a fine-tuned fastconformer-ctc-xl with Serialized Output Training achieving leading cpWER (e.g., 6.97% cpWER on the test set). The dataset is publicly available on Hugging Face, offering a reproducible, scalable resource for advancing multi-speaker speech processing in controlled yet realistic conversational settings.

Abstract

We introduce LibriConvo, a simulated multi-speaker conversational dataset based on speaker-aware conversation simulation (SASC), designed to support training and evaluation of speaker diarization and automatic speech recognition (ASR) systems. Unlike prior resources that mostly rely on semantically disconnected utterances and implausible temporal gaps, LibriConvo ensures semantic coherence and realistic conversational timing. Our pipeline leverages CallHome with external VAD for reliable boundaries, applies compression to reduce unnaturally long silences, and organizes LibriTTS utterances by book to maintain contextual consistency. Acoustic realism is enhanced via a novel room impulse response selection procedure that ranks speaker-microphone configurations by spatial plausibility, balancing realism and diversity. The dataset comprises 240.1 hours across 1,496 dialogues with 830 unique speakers, split in a speaker-disjoint manner for robust evaluation. Baselines show that the sortformer model outperforms the pyannote pipeline in diarization, while a fine-tuned Fast Conformer-CTC XLarge with Serialized Output Training achieves 7.29\% WER for ASR, surpassing zero-shot Whisper-large-v3. LibriConvo provides a valuable resource for advancing multi-speaker speech processing research with realistic conversational dynamics and controlled experimental conditions.
Paper Structure (15 sections, 1 equation, 2 figures, 3 tables, 1 algorithm)

This paper contains 15 sections, 1 equation, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Speaker-aware conversation simulation
  4. Dataset generation
  5. Statistics
  6. Utterances
  7. Room impulse responses
  8. Creating splits
  9. Experiments
  10. Data preparation
  11. Diarization
  12. ASR
  13. Conclusion
  14. Bibliographical References
  15. Language Resource References

Figures (2)

  • Figure 1: Comparison between original CallHome annotations and VAD-derived speech segments.
  • Figure 2: Distribution of DER values across test recordings.