Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmoAra: Emotion-Preserving English Speech Transcription and Cross-Lingual Translation with Arabic Text-to-Speech

Besher Hassan, Ibrahim Alsarraj, Musaab Hasan, Yousef Melhim, Shahem Fadi, Shahem Sultan

TL;DR

The paper tackles the challenge of cross-lingual spoken communication in banking by preserving emotional content across language barriers. It introduces EmoAra, an end-to-end pipeline that combines CNN-based speech emotion recognition, Whisper ASR, a fine-tuned MarianMT translation model, and MMS-TTS-Ara speech synthesis to transcribe English speech, translate to Arabic, and generate Arabic speech with preserved emotion. Key results show a 0.94 F1-score for emotion classification, a BLEU score of 56 and a BERTScore of 88.7% for translation, and an 81% human evaluation score on banking-domain translations, demonstrating effective domain adaptation and emotion retention. The work highlights the practical potential of emotion-aware multilingual pipelines for customer service in finance and points to future gains from data expansion and advanced fine-tuning techniques.

Abstract

This work presents EmoAra, an end-to-end emotion-preserving pipeline for cross-lingual spoken communication, motivated by banking customer service where emotional context affects service quality. EmoAra integrates Speech Emotion Recognition, Automatic Speech Recognition, Machine Translation, and Text-to-Speech to process English speech and deliver an Arabic spoken output while retaining emotional nuance. The system uses a CNN-based emotion classifier, Whisper for English transcription, a fine-tuned MarianMT model for English-to-Arabic translation, and MMS-TTS-Ara for Arabic speech synthesis. Experiments report an F1-score of 94% for emotion classification, translation performance of BLEU 56 and BERTScore F1 88.7%, and an average human evaluation score of 81% on banking-domain translations. The implementation and resources are available at the accompanying GitHub repository.

EmoAra: Emotion-Preserving English Speech Transcription and Cross-Lingual Translation with Arabic Text-to-Speech

TL;DR

The paper tackles the challenge of cross-lingual spoken communication in banking by preserving emotional content across language barriers. It introduces EmoAra, an end-to-end pipeline that combines CNN-based speech emotion recognition, Whisper ASR, a fine-tuned MarianMT translation model, and MMS-TTS-Ara speech synthesis to transcribe English speech, translate to Arabic, and generate Arabic speech with preserved emotion. Key results show a 0.94 F1-score for emotion classification, a BLEU score of 56 and a BERTScore of 88.7% for translation, and an 81% human evaluation score on banking-domain translations, demonstrating effective domain adaptation and emotion retention. The work highlights the practical potential of emotion-aware multilingual pipelines for customer service in finance and points to future gains from data expansion and advanced fine-tuning techniques.

Abstract

This work presents EmoAra, an end-to-end emotion-preserving pipeline for cross-lingual spoken communication, motivated by banking customer service where emotional context affects service quality. EmoAra integrates Speech Emotion Recognition, Automatic Speech Recognition, Machine Translation, and Text-to-Speech to process English speech and deliver an Arabic spoken output while retaining emotional nuance. The system uses a CNN-based emotion classifier, Whisper for English transcription, a fine-tuned MarianMT model for English-to-Arabic translation, and MMS-TTS-Ara for Arabic speech synthesis. Experiments report an F1-score of 94% for emotion classification, translation performance of BLEU 56 and BERTScore F1 88.7%, and an average human evaluation score of 81% on banking-domain translations. The implementation and resources are available at the accompanying GitHub repository.
Paper Structure (33 sections, 1 equation, 3 figures, 6 tables)

This paper contains 33 sections, 1 equation, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Project Overview
  3. Primary Contributions
  4. Related Work
  5. Problem Statement
  6. Proposed Method
  7. Exploratory Data Analysis
  8. Translation Data Preprocessing
  9. Audio Files
  10. Method
  11. CNN Model for Speech Emotion Recognition:
  12. Whisper by OpenAI
  13. English To Arabic Translation
  14. Text-to-Speech Using MMS-TTS-Ara
  15. Experimental Setup
  16. ...and 18 more sections

Figures (3)

  • Figure 1: Log-Frequency Power Spectrum of Different Emotions
  • Figure 2: System Architecture
  • Figure 3: Top 5 Words in English and Arabic Dataset