Enhancing Non-Core Language Instruction-Following in Speech LLMs via Semi-Implicit Cross-Lingual CoT Reasoning

TL;DR

This work tackles the challenge of non-core language instruction-following in speech LLMs by embedding speech-to-text translation within the model's reasoning through a Cross-lingual Speech CoT (XS-CoT) framework. It introduces four token types to enable cross-lingual transfer of reasoning from core languages to non-core languages and employs a Semi-Implicit CoT strategy to compress intermediate reasoning tokens, reducing latency while preserving reasoning structure. Empirical results across SALMONN and Qwen2Audio show up to ~45% relative GPT-4 score improvements for non-core languages and substantial latency reductions, with modest data needs due to leveraging core-language reasoning. An open-source data pipeline and datasets for Japanese, German, and French support training and replication, advancing practical multilingual speech instruction-following capabilities in SLLMs.

Abstract

Large language models have been extended to the speech domain, leading to the development of speech large language models (SLLMs). While existing SLLMs demonstrate strong performance in speech instruction-following for core languages (e.g., English), they often struggle with non-core languages due to the scarcity of paired speech-text data and limited multilingual semantic reasoning capabilities. To address this, we propose the semi-implicit Cross-lingual Speech Chain-of-Thought (XS-CoT) framework, which integrates speech-to-text translation into the reasoning process of SLLMs. The XS-CoT generates four types of tokens: instruction and response tokens in both core and non-core languages, enabling cross-lingual transfer of reasoning capabilities. To mitigate inference latency in generating target non-core response tokens, we incorporate a semi-implicit CoT scheme into XS-CoT, which progressively compresses the first three types of intermediate reasoning tokens while retaining global reasoning logic during training. By leveraging the robust reasoning capabilities of the core language, XS-CoT improves responses for non-core languages by up to 45\% in GPT-4 score when compared to direct supervised fine-tuning on two representative SLLMs, Qwen2-Audio and SALMONN. Moreover, the semi-implicit XS-CoT reduces token delay by more than 50\% with a slight drop in GPT-4 scores. Importantly, XS-CoT requires only a small amount of high-quality training data for non-core languages by leveraging the reasoning capabilities of core languages. To support training, we also develop a data pipeline and open-source speech instruction-following datasets in Japanese, German, and French.

Figures (6)

• Figure 1: An overview of XS-CoT SLLM framework: speech instruction as input, text tokens as output, with red for target non-core languages (ja) and green for core languages (en).
• Figure 2: Example responses: direct SFT target language (ja) output (top) vs. XS-CoT output (bottom). The words in blue are translations made for ease of understanding and are not the output of SLLM.
• Figure 3: Stepwise internalization for semi-implicit XS-CoT reasoning in core language response token. The process progressively reduces the number of tokens during training, thereby decreasing inference latency.
• Figure 4: Overview of the data pipeline for generating four types of tokens. 'en' stands for English, 'ja' stands for the target language Japanese.
• Figure 5: Impact of the hyperparameter $k$ on the Semi-Implicit CoT method. The horizontal axis represents different $k$ values (1, 3, 5, 7, and full reasoning chain), with the top sub-graphs displaying GPT-4 scores and the bottom sub-graphs showing the average generated CoT token lengths.