Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selection

Hsi-Che Lin, Yi-Cheng Lin, Huang-Cheng Chou, Hung-yi Lee

TL;DR

This work tackles the challenge of scarce labeled Speech Emotion Recognition (SER) data in under-resourced languages by combining expressive Speech-to-Speech Translation (S2ST) with a bootstrapping data-selection pipeline to generate and curate target-language SER data. The method translates high-resource-language SER samples into the target language, filters translations, and iteratively selects the most distribution-matching synthetic samples to train robust models, testing across multiple upstream SER architectures and datasets. Across diverse languages and models, the approach yields consistent performance gains, especially on weaker datasets, and demonstrates the importance of data expressivity and principled sample selection. The study also offers practical guidelines for choosing source datasets and highlights directions for adaptive bootstrapping to further improve scalability in multilingual SER.

Abstract

Speech Emotion Recognition (SER) is a crucial component in developing general-purpose AI agents capable of natural human-computer interaction. However, building robust multilingual SER systems remains challenging due to the scarcity of labeled data in languages other than English and Chinese. In this paper, we propose an approach to enhance SER performance in low SER resource languages by leveraging data from high-resource languages. Specifically, we employ expressive Speech-to-Speech translation (S2ST) combined with a novel bootstrapping data selection pipeline to generate labeled data in the target language. Extensive experiments demonstrate that our method is both effective and generalizable across different upstream models and languages. Our results suggest that this approach can facilitate the development of more scalable and robust multilingual SER systems.

Improving Speech Emotion Recognition in Under-Resourced Languages via Speech-to-Speech Translation with Bootstrapping Data Selection

TL;DR

This work tackles the challenge of scarce labeled Speech Emotion Recognition (SER) data in under-resourced languages by combining expressive Speech-to-Speech Translation (S2ST) with a bootstrapping data-selection pipeline to generate and curate target-language SER data. The method translates high-resource-language SER samples into the target language, filters translations, and iteratively selects the most distribution-matching synthetic samples to train robust models, testing across multiple upstream SER architectures and datasets. Across diverse languages and models, the approach yields consistent performance gains, especially on weaker datasets, and demonstrates the importance of data expressivity and principled sample selection. The study also offers practical guidelines for choosing source datasets and highlights directions for adaptive bootstrapping to further improve scalability in multilingual SER.

Abstract

Speech Emotion Recognition (SER) is a crucial component in developing general-purpose AI agents capable of natural human-computer interaction. However, building robust multilingual SER systems remains challenging due to the scarcity of labeled data in languages other than English and Chinese. In this paper, we propose an approach to enhance SER performance in low SER resource languages by leveraging data from high-resource languages. Specifically, we employ expressive Speech-to-Speech translation (S2ST) combined with a novel bootstrapping data selection pipeline to generate labeled data in the target language. Extensive experiments demonstrate that our method is both effective and generalizable across different upstream models and languages. Our results suggest that this approach can facilitate the development of more scalable and robust multilingual SER systems.
Paper Structure (20 sections, 3 equations, 2 figures, 6 tables)

This paper contains 20 sections, 3 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem definition
  4. Phase 1 - Data synthesis
  5. Phase 2 - Bootstrapping
  6. Selection criterion for bootstrapping
  7. Experiment setup
  8. Datasets
  9. Speech Translation
  10. Upstream model
  11. Evaluation protocol
  12. Result and Discussion
  13. Main result
  14. Comparison to other methods
  15. Ablation study
  16. ...and 5 more sections

Figures (2)

  • Figure 1: Overview of our method.
  • Figure 2: Effect of different numbers of iterations for bootstrapping on EmoDB and EMOVO. The dashed line is the oracle result where we select the best step in each fold and seed.