Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

Kirill Borodin, Vasiliy Kudryavtsev, Maxim Maslov, Nikita Vasiliev, Mikhail Gorodnichev, Grach Mkrtchian

Abstract

We investigate multi-stage pretraining for prosody modeling in diffusion-based TTS. A speaker-conditioned dual-stream encoder is trained with masked language modeling followed by SigLIP-style cross-modal contrastive learning using mixed-phoneme batches, with an additional same-phoneme refinement stage studied separately. We evaluate intrinsic text-audio retrieval and downstream synthesis in Grad-TTS and a latent diffusion TTS system. The two-stage curriculum (MLM + mixed-phoneme contrastive learning) achieves the best overall synthesis quality in terms of intelligibility, speaker similarity, and perceptual measures. Although same-phoneme refinement improves prosodic retrieval, it reduces phoneme discrimination and degrades synthesis. These findings indicate that improvements in embedding-space metrics do not necessarily translate to better generative performance and highlight the need to balance phoneme discrimination and prosodic sensitivity in TTS pretraining.

Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

Abstract

We investigate multi-stage pretraining for prosody modeling in diffusion-based TTS. A speaker-conditioned dual-stream encoder is trained with masked language modeling followed by SigLIP-style cross-modal contrastive learning using mixed-phoneme batches, with an additional same-phoneme refinement stage studied separately. We evaluate intrinsic text-audio retrieval and downstream synthesis in Grad-TTS and a latent diffusion TTS system. The two-stage curriculum (MLM + mixed-phoneme contrastive learning) achieves the best overall synthesis quality in terms of intelligibility, speaker similarity, and perceptual measures. Although same-phoneme refinement improves prosodic retrieval, it reduces phoneme discrimination and degrades synthesis. These findings indicate that improvements in embedding-space metrics do not necessarily translate to better generative performance and highlight the need to balance phoneme discrimination and prosodic sensitivity in TTS pretraining.

Paper Structure

This paper contains 19 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Prosody Encoder Architecture
  4. Training Curriculum
  5. Stage 1: Masked Language Modelling.
  6. Stage 2: Cross-Modal Phoneme Discrimination.
  7. Stage 3: CLAPSpeech-Style Same-Phoneme Refinement.
  8. Experimental Setup
  9. Dataset
  10. Encoder Training
  11. Downstream Models
  12. Evaluation Metrics
  13. Results and Analysis
  14. RQ1: Curriculum Stage Contributions
  15. RQ2: Do Intrinsic Gains Transfer to Synthesis?
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Prosody encoder architecture. Phoneme and BPE streams are independently embedded and processed by speaker-conditioned transformer encoders. BPE hidden states are aggregated via word-level pooling and expanded to phoneme resolution. The fused streams pass through a shared encoder, layer normalization, and a convolutional projection to produce per-phoneme prosodic embeddings.