Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Classification of Spontaneous and Scripted Speech for Multilingual Audio

Shahar Elisha, Andrew McDowell, Mariano Beguerisse-Díaz, Emmanouil Benetos

TL;DR

Distinguishing scripted from spontaneous speech in multilingual audio is crucial for scalable media understanding and robust speech processing. The authors systematically compare handcrafted acoustic features with transformer-based representations, using a large multilingual podcast dataset and cross-domain tests (CEFC, DIHARD) to assess generalization. Whisper-based transformers achieve state-of-the-art performance across languages and domains, though some languages (e.g., Japanese) exhibit notable biases, suggesting language conditioning or language-specific models as potential remedies. The study provides practical guidance for content labeling and model evaluation in diverse linguistic contexts, while highlighting ongoing challenges in underrepresented languages and cross-domain variability.

Abstract

Distinguishing scripted from spontaneous speech is an essential tool for better understanding how speech styles influence speech processing research. It can also improve recommendation systems and discovery experiences for media users through better segmentation of large recorded speech catalogues. This paper addresses the challenge of building a classifier that generalises well across different formats and languages. We systematically evaluate models ranging from traditional, handcrafted acoustic and prosodic features to advanced audio transformers, utilising a large, multilingual proprietary podcast dataset for training and validation. We break down the performance of each model across 11 language groups to evaluate cross-lingual biases. Our experimental analysis extends to publicly available datasets to assess the models' generalisability to non-podcast domains. Our results indicate that transformer-based models consistently outperform traditional feature-based techniques, achieving state-of-the-art performance in distinguishing between scripted and spontaneous speech across various languages.

Classification of Spontaneous and Scripted Speech for Multilingual Audio

TL;DR

Distinguishing scripted from spontaneous speech in multilingual audio is crucial for scalable media understanding and robust speech processing. The authors systematically compare handcrafted acoustic features with transformer-based representations, using a large multilingual podcast dataset and cross-domain tests (CEFC, DIHARD) to assess generalization. Whisper-based transformers achieve state-of-the-art performance across languages and domains, though some languages (e.g., Japanese) exhibit notable biases, suggesting language conditioning or language-specific models as potential remedies. The study provides practical guidance for content labeling and model evaluation in diverse linguistic contexts, while highlighting ongoing challenges in underrepresented languages and cross-domain variability.

Abstract

Distinguishing scripted from spontaneous speech is an essential tool for better understanding how speech styles influence speech processing research. It can also improve recommendation systems and discovery experiences for media users through better segmentation of large recorded speech catalogues. This paper addresses the challenge of building a classifier that generalises well across different formats and languages. We systematically evaluate models ranging from traditional, handcrafted acoustic and prosodic features to advanced audio transformers, utilising a large, multilingual proprietary podcast dataset for training and validation. We break down the performance of each model across 11 language groups to evaluate cross-lingual biases. Our experimental analysis extends to publicly available datasets to assess the models' generalisability to non-podcast domains. Our results indicate that transformer-based models consistently outperform traditional feature-based techniques, achieving state-of-the-art performance in distinguishing between scripted and spontaneous speech across various languages.

Paper Structure

This paper contains 18 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Experimental setup
  4. Data
  5. Podcast dataset
  6. Publicly available datasets
  7. Models
  8. Handcrafted features
  9. YAMNet
  10. Whisper
  11. Neural network
  12. Aggregation
  13. Evaluation
  14. Results
  15. Overall performance
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Distribution of spontaneous and scripted files in the podcast dataset split by language.
  • Figure 2: Distribution of median episode-level scores for scripted and spontaneous per feature model. Scores towards 1 are predicted as 'Scripted' and towards 0 as 'Spontaneous'.
  • Figure 3: AUC computed on median episode-level scores for each feature model split by language.