Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings

Praveen Srinivasa Varadhan, Ashwin Sankar, Giri Raju, Mitesh M. Khapra

TL;DR

This work addresses the scarcity of expressive TTS data for Indian languages by introducing Rasa, a multilingual expressive TTS dataset for Assamese, Bengali, and Tamil with 10 hours of neutral speech and 1–3 hours of expressive speech per emotion across six Ekman emotions. The authors combine LLM-assisted sentence generation, scenario-based human writing, and neutral sentence reuse to create a diverse, emotion-rich corpus, followed by professional studio recordings and rigorous quality control. They train expressive TTS models using FastPitch with unsupervised alignment and emotion-conditioned encodings, paired with HiFiGAN for audio synthesis, and conduct thorough ablations to identify data requirements, finding that 1 hour of neutral data plus 30 minutes of expressive data per emotion can yield Fair expressiveness, with more neutral data further boosting performance. The approach generalizes to Assamese, Bengali, and Hindi and provides practical guidance for collecting balanced neutral and expressive data in low-resource settings, with the promise of broad impact for research and applications in Indian languages.”

Abstract

We release Rasa, the first multilingual expressive TTS dataset for any Indian language, which contains 10 hours of neutral speech and 1-3 hours of expressive speech for each of the 6 Ekman emotions covering 3 languages: Assamese, Bengali, & Tamil. Our ablation studies reveal that just 1 hour of neutral and 30 minutes of expressive data can yield a Fair system as indicated by MUSHRA scores. Increasing neutral data to 10 hours, with minimal expressive data, significantly enhances expressiveness. This offers a practical recipe for resource-constrained languages, prioritizing easily obtainable neutral data alongside smaller amounts of expressive data. We show the importance of syllabically balanced data and pooling emotions to enhance expressiveness. We also highlight challenges in generating specific emotions, e.g., fear and surprise.

Rasa: Building Expressive Speech Synthesis Systems for Indian Languages in Low-resource Settings

TL;DR

This work addresses the scarcity of expressive TTS data for Indian languages by introducing Rasa, a multilingual expressive TTS dataset for Assamese, Bengali, and Tamil with 10 hours of neutral speech and 1–3 hours of expressive speech per emotion across six Ekman emotions. The authors combine LLM-assisted sentence generation, scenario-based human writing, and neutral sentence reuse to create a diverse, emotion-rich corpus, followed by professional studio recordings and rigorous quality control. They train expressive TTS models using FastPitch with unsupervised alignment and emotion-conditioned encodings, paired with HiFiGAN for audio synthesis, and conduct thorough ablations to identify data requirements, finding that 1 hour of neutral data plus 30 minutes of expressive data per emotion can yield Fair expressiveness, with more neutral data further boosting performance. The approach generalizes to Assamese, Bengali, and Hindi and provides practical guidance for collecting balanced neutral and expressive data in low-resource settings, with the promise of broad impact for research and applications in Indian languages.”

Abstract

We release Rasa, the first multilingual expressive TTS dataset for any Indian language, which contains 10 hours of neutral speech and 1-3 hours of expressive speech for each of the 6 Ekman emotions covering 3 languages: Assamese, Bengali, & Tamil. Our ablation studies reveal that just 1 hour of neutral and 30 minutes of expressive data can yield a Fair system as indicated by MUSHRA scores. Increasing neutral data to 10 hours, with minimal expressive data, significantly enhances expressiveness. This offers a practical recipe for resource-constrained languages, prioritizing easily obtainable neutral data alongside smaller amounts of expressive data. We show the importance of syllabically balanced data and pooling emotions to enhance expressiveness. We also highlight challenges in generating specific emotions, e.g., fear and surprise.
Paper Structure (20 sections, 1 figure, 7 tables)

This paper contains 20 sections, 1 figure, 7 tables.

Table of Contents

  1. Introduction
  2. Rasa: A High-Quality Expressive Speech Synthesis Dataset
  3. Generating Recording Scripts
  4. LLM-Assisted Corpora Creation
  5. Scenario-prompted Human-Assisted Corpora Creation
  6. Reusing Existing Corpora for Neutral Sentences
  7. Recording Audio
  8. Quality Control
  9. Dataset Statistics
  10. Methodology
  11. Model Architecture
  12. Syllabic-Balanced Text Selection
  13. Experimental Setup
  14. Results
  15. Evaluation of Low-Resource Neutral TTS
  16. ...and 5 more sections

Figures (1)

  • Figure 1: Confusion plots for LR (left) and HR (right) model.