TTA: Transcribe, Translate and Alignment for Cross-lingual Speech Representation

TL;DR

This work introduces TTA, a lightweight cross-lingual speech foundation model designed to enhance integration with large language models. By fusing a Zipformer-based Transducer with an attention-based AED and a frozen multilingual BERT alignment module, TTA learns joint MASR, ST, and cross-lingual alignment through a SigLIP contrastive loss on $H$ and $T$. The approach demonstrates superior MASR/ST performance and strong cross-lingual retrieval while maintaining a small parameter footprint, and reveals nuanced interactions between cross-lingual representations and ASR/ST tasks. The authors also show that explicit semantic alignment improves downstream ASR-LLM performance and plan to release model weights and recipes via the Auden toolkit, enabling reproducibility and broader adoption.

Abstract

Speech-LLM models have demonstrated great performance in multi-modal and multi-task speech understanding. A typical speech-LLM paradigm is integrating speech modality with a large language model (LLM). While the Whisper encoder was frequently adopted in previous studies for speech input, it shows limitations regarding input format, model scale, and semantic performance. To this end, we propose a lightweight TTA model specialized in speech semantics for more effective LLM integration. With large-scale training of 358k hours of speech data on multilingual speech recognition (ASR), speech translation (ST) and speech-text alignment tasks, TTA is capable of producing robust cross-lingual speech representations. Extensive evaluations across diverse benchmarks, including ASR/ST, speech retrieval, and ASR-LLM performance assessments, demonstrate TTA's superiority over Whisper. Furthermore, we rigorously validate the interplay between cross-lingual capabilities and ASR/ST performance. The model weights and training recipes of TTA will be released as part of an audio understanding toolkit Auden.

Figures (4)

• Figure 1: The overall diagram of TTA model.
• Figure 2: Statistics of ASR training data over 10 languages.
• Figure 3: Heatmaps of speech-to-speech retrieval accuracy across 10 languages. Inside each subfigure, each cell represent the retrieval accuracy from language X to language Y. The deeper color indicates higher retrieval accuracy and better cross-lingual alignment.
• Figure 4: Two linear probing experiments. (a) compares averaged validation losses of the speech translation probing task with different encoders on CoVoSTv2. (b) shows the recognition accuracy curves, comparing various encoders in the ASR-LLM training using Aishell2 and Librispeech.