AdaST: Dynamically Adapting Encoder States in the Decoder for End-to-End Speech-to-Text Translation
Wuwei Huang, Dexin Wang, Deyi Xiong
TL;DR
AdaST tackles the limitation of fixed encoder representations in end-to-end speech translation by dynamically adapting acoustic states within the decoder. It introduces a unified decoder that concatenates acoustic states with target embeddings and utilizes a speech-text mixed attention mechanism, along with modality embeddings, to enable deep cross-modal interactions in a shared space. Empirical results on IWSLT18 En-De and Augmented LibriSpeech En-Fr show BLEU gains of up to +1.18 over strong baselines, with AdaST using fewer parameters and modest latency overhead. Analyses reveal that adaptive acoustic states improve cross-modal/cross-lingual translation and that the encoder shifts toward encoding semantic content, while the decoder handles cross-modal integration, highlighting practical benefits for end-to-end ST systems.
Abstract
In end-to-end speech translation, acoustic representations learned by the encoder are usually fixed and static, from the perspective of the decoder, which is not desirable for dealing with the cross-modal and cross-lingual challenge in speech translation. In this paper, we show the benefits of varying acoustic states according to decoder hidden states and propose an adaptive speech-to-text translation model that is able to dynamically adapt acoustic states in the decoder. We concatenate the acoustic state and target word embedding sequence and feed the concatenated sequence into subsequent blocks in the decoder. In order to model the deep interaction between acoustic states and target hidden states, a speech-text mixed attention sublayer is introduced to replace the conventional cross-attention network. Experiment results on two widely-used datasets show that the proposed method significantly outperforms state-of-the-art neural speech translation models.