Investigating Decoder-only Large Language Models for Speech-to-text Translation
Chao-Wei Huang, Hui Lu, Hongyu Gong, Hirofumi Inaguma, Ilia Kulikov, Ruslan Mavlyutov, Sravya Popuri
TL;DR
This work investigates integrating decoder-only large language models (LLMs) into speech-to-text translation (S2TT) by feeding continuous speech representations into a decoder-only architecture. The proposed approach combines a W2v-BERT speech encoder, a length adapter, and a decoder such as LLaMA-2 to produce translations without discretizing speech, trained with multiple task formulations and parameter-efficient fine-tuning. Results show state-of-the-art BLEU on CoVoST2 and competitive performance on FLEURS using only public data, outperforming several proprietary baselines and providing strong evidence that decoder-only LLMs can effectively handle S2TT with careful architectural and training choices. The work also demonstrates that LNA-based fine-tuning outperforms LoRA for this task, and provides practical guidance on formulation, architecture, and PEFT design for future S2TT research with LLMs.
Abstract
Large language models (LLMs), known for their exceptional reasoning capabilities, generalizability, and fluency across diverse domains, present a promising avenue for enhancing speech-related tasks. In this paper, we focus on integrating decoder-only LLMs to the task of speech-to-text translation (S2TT). We propose a decoder-only architecture that enables the LLM to directly consume the encoded speech representation and generate the text translation. Additionally, we investigate the effects of different parameter-efficient fine-tuning techniques and task formulation. Our model achieves state-of-the-art performance on CoVoST 2 and FLEURS among models trained without proprietary data. We also conduct analyses to validate the design choices of our proposed model and bring insights to the integration of LLMs to S2TT.