Finetuning End-to-End Models for Estonian Conversational Spoken Language Translation

TL;DR

The paper tackles the scarcity of Estonian conversational speech translation data by finetuning end-to-end multilingual SLT models (Whisper, OWSM 3.1 EBF, SeamlessM4T) using synthetic MT-based data and web-scraped transcripts. It compares cascaded and end-to-end approaches, finds that synthetic-data finetuning substantially boosts translation accuracy, and identifies SeamlessM4T as the most robust performer across Estonian-English and Estonian-Russian directions, sometimes surpassing cascaded baselines. An evaluation set of realistic conversational speech is released, along with public finetuned models, underscoring practical viability for low-resource languages. The work highlights the viability of repurposing Whisper for non-English directions via careful finetuning and underscores the importance of diverse, conversational data for robust SLT.

Abstract

This paper investigates the finetuning of end-to-end models for bidirectional Estonian-English and Estonian-Russian conversational speech-to-text translation. Due to the limited availability of speech translation data for Estonian, we created additional training data by web scraping and synthesizing data from speech recognition datasets using machine translation. We evaluated three publicly available end-to-end models: Whisper, OWSM 3.1, and SeamlessM4T. Our results indicate that fine-tuning with synthetic data enhances translation accuracy by a large margin, with SeamlessM4T matching or surpassing cascaded speech translation systems that use state-of-the-art speech recognition and machine translation models.