Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speech

Haowei Lou, Hye-young Paik, Wen Hu, Lina Yao

TL;DR

ParaMETA addresses entangled paralinguistic representations by learning a structured, disentangled embedding framework: a shared META space regularized by graded contrastive learning, followed by per-task subspaces with prototype-based supervision, and an efficient text–speech alignment via projection. This design reduces inter-task interference and negative transfer while enabling both accurate speaking-style classification and controllable TTS through speech and text prompts. Empirical results show superior classification performance across multiple backbones, robust style manipulation, and perceptually improved TTS quality, with significantly lower model footprint and memory requirements than CLAP-based baselines. The approach offers practical impact for real-world HCI, multilingual synthesis, and flexible style-controlled generation in resource-constrained settings.

Abstract

Learning representative embeddings for different types of speaking styles, such as emotion, age, and gender, is critical for both recognition tasks (e.g., cognitive computing and human-computer interaction) and generative tasks (e.g., style-controllable speech generation). In this work, we introduce ParaMETA, a unified and flexible framework for learning and controlling speaking styles directly from speech. Unlike existing methods that rely on single-task models or cross-modal alignment, ParaMETA learns disentangled, task-specific embeddings by projecting speech into dedicated subspaces for each type of style. This design reduces inter-task interference, mitigates negative transfer, and allows a single model to handle multiple paralinguistic tasks such as emotion, gender, age, and language classification. Beyond recognition, ParaMETA enables fine-grained style control in Text-To-Speech (TTS) generative models. It supports both speech- and text-based prompting and allows users to modify one speaking styles while preserving others. Extensive experiments demonstrate that ParaMETA outperforms strong baselines in classification accuracy and generates more natural and expressive speech, while maintaining a lightweight and efficient model suitable for real-world applications.

ParaMETA: Towards Learning Disentangled Paralinguistic Speaking Styles Representations from Speech

TL;DR

ParaMETA addresses entangled paralinguistic representations by learning a structured, disentangled embedding framework: a shared META space regularized by graded contrastive learning, followed by per-task subspaces with prototype-based supervision, and an efficient text–speech alignment via projection. This design reduces inter-task interference and negative transfer while enabling both accurate speaking-style classification and controllable TTS through speech and text prompts. Empirical results show superior classification performance across multiple backbones, robust style manipulation, and perceptually improved TTS quality, with significantly lower model footprint and memory requirements than CLAP-based baselines. The approach offers practical impact for real-world HCI, multilingual synthesis, and flexible style-controlled generation in resource-constrained settings.

Abstract

Learning representative embeddings for different types of speaking styles, such as emotion, age, and gender, is critical for both recognition tasks (e.g., cognitive computing and human-computer interaction) and generative tasks (e.g., style-controllable speech generation). In this work, we introduce ParaMETA, a unified and flexible framework for learning and controlling speaking styles directly from speech. Unlike existing methods that rely on single-task models or cross-modal alignment, ParaMETA learns disentangled, task-specific embeddings by projecting speech into dedicated subspaces for each type of style. This design reduces inter-task interference, mitigates negative transfer, and allows a single model to handle multiple paralinguistic tasks such as emotion, gender, age, and language classification. Beyond recognition, ParaMETA enables fine-grained style control in Text-To-Speech (TTS) generative models. It supports both speech- and text-based prompting and allows users to modify one speaking styles while preserving others. Extensive experiments demonstrate that ParaMETA outperforms strong baselines in classification accuracy and generates more natural and expressive speech, while maintaining a lightweight and efficient model suitable for real-world applications.
Paper Structure (18 sections, 11 equations, 2 figures, 4 tables)

This paper contains 18 sections, 11 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Speech Encoder
  4. META Embedding
  5. Task-Specific Embedding
  6. Prototype Learning
  7. Text and Speech Alignment
  8. Downstream Application
  9. Speaking Styles Classification.
  10. Style Controllable TTS.
  11. Speaking Styles Manipulation.
  12. Experiments
  13. Dataset and Experimental Setup
  14. Speaking Styles Classification
  15. Style-Controllable TTS Generation
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Overview of the ParaMETA framework. Given a speech sample and its style caption, ParaMETA decomposes the caption into $T$ tasks and encodes the speech into a META space with regularization. The META embedding is projected into task-specific subspaces, where contrastive and prototype alignment losses structure the representation. The learned embeddings support downstream applications such as classification, style controllable TTS, and style manipulation via both speech and text prompts.
  • Figure 2: Embedding Visualization