Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understanding

Runchuan Ye, Yixuan Zhou, Renjie Yu, Zijian Lin, Kehan Li, Xiang Li, Xin Liu, Guoyang Zeng, Zhiyong Wu

TL;DR

The paper addresses the scarcity of non-verbal vocalizations (NVs) in current speech systems by introducing a fully automated annotation pipeline and a large-scale NV dataset, NonVerbalSpeech-38K. It combines a unified frame-level NV detector with a Temporal-Semantic Alignment (TSA) module to label NVs in in-the-wild speech and integrate NV tags into transcripts. The resulting NVS-38K dataset, spanning 131 hours and 10 NV types in English and Chinese, enables effective NV generation and understanding, with the TSA-based approach delivering superior NV controllability and improved NV transcription. This scalable framework reduces manual labeling, preserves natural NV sounds, and supports cross-lingual NV research, offering a practical resource for advancing expressive speech technologies.

Abstract

Non-verbal Vocalizations (NVs), such as laughter and sighs, are vital for conveying emotion and intention in human speech, yet most existing speech systems neglect them, which severely compromises communicative richness and emotional intelligence. Existing methods for NVs acquisition are either costly and unscalable (relying on manual annotation/recording) or unnatural (relying on rule-based synthesis). To address these limitations, we propose a highly scalable automatic annotation framework to label non-verbal phenomena from natural speech, which is low-cost, easily extendable, and inherently diverse and natural. This framework leverages a unified detection model to accurately identify NVs in natural speech and integrates them with transcripts via temporal-semantic alignment method. Using this framework, we created and released \textbf{NonVerbalSpeech-38K}, a diverse, real-world dataset featuring 38,718 samples across 10 NV categories collected from in-the-wild media. Experimental results demonstrate that our dataset provides superior controllability for NVs generation and achieves comparable performance for NVs understanding.

A Scalable Pipeline for Enabling Non-Verbal Speech Generation and Understanding

TL;DR

The paper addresses the scarcity of non-verbal vocalizations (NVs) in current speech systems by introducing a fully automated annotation pipeline and a large-scale NV dataset, NonVerbalSpeech-38K. It combines a unified frame-level NV detector with a Temporal-Semantic Alignment (TSA) module to label NVs in in-the-wild speech and integrate NV tags into transcripts. The resulting NVS-38K dataset, spanning 131 hours and 10 NV types in English and Chinese, enables effective NV generation and understanding, with the TSA-based approach delivering superior NV controllability and improved NV transcription. This scalable framework reduces manual labeling, preserves natural NV sounds, and supports cross-lingual NV research, offering a practical resource for advancing expressive speech technologies.

Abstract

Non-verbal Vocalizations (NVs), such as laughter and sighs, are vital for conveying emotion and intention in human speech, yet most existing speech systems neglect them, which severely compromises communicative richness and emotional intelligence. Existing methods for NVs acquisition are either costly and unscalable (relying on manual annotation/recording) or unnatural (relying on rule-based synthesis). To address these limitations, we propose a highly scalable automatic annotation framework to label non-verbal phenomena from natural speech, which is low-cost, easily extendable, and inherently diverse and natural. This framework leverages a unified detection model to accurately identify NVs in natural speech and integrates them with transcripts via temporal-semantic alignment method. Using this framework, we created and released \textbf{NonVerbalSpeech-38K}, a diverse, real-world dataset featuring 38,718 samples across 10 NV categories collected from in-the-wild media. Experimental results demonstrate that our dataset provides superior controllability for NVs generation and achieves comparable performance for NVs understanding.

Paper Structure

This paper contains 35 sections, 4 equations, 5 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Methodology
  3. Data Preparation and Preprocessing
  4. Non-Verbal Segments Detection
  5. Model Architecture
  6. Data Preparation for Model Training
  7. Experimental Setup and Results
  8. Non-Verbal Region Detection Using the Trained Frame-Level Detection Model
  9. Incorporating Non-Verbal Tags into Speech Content
  10. The NonVerbalSpeech-38K Dataset
  11. Experiments
  12. Experimental Setup for Non-Verbal Speech Generation
  13. Implementation Details
  14. Baselines
  15. Evaluation
  16. ...and 20 more sections

Figures (5)

  • Figure 1: (a) The task definition, (b) The proposed frame-level detection model, and (c) The TSA method used in the pipeline (Fig. \ref{['fig:pipeline']}).
  • Figure 2: The proposed NonVerbalSpeech-38K data processing pipeline. Colors other than black in the waveform indicate non-verbal vocalizations.
  • Figure 3: Frame-level detection model test results and distribution of training NV segments.
  • Figure 4: Language and NV label distribution in proposed NVS dataset.
  • Figure 5: Speech duration distribution of the proposed NonVerbalSpeech-38K dataset.