AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning

TL;DR

AZeroS introduces Self-Generated Instruction-Free Tuning (SIFT) to extend LLMs to speech without task-specific instruction data, training only lightweight projection modules while keeping the LLM and encoders frozen. Built on Qwen2.5-7B-Instruct, AZeroS achieves strong semantic and paralinguistic understanding, approaching text-only semantic upper bounds on VoiceBench and outperforming cascaded and end-to-end baselines on AIR-Bench. The method uses a two-stage training regime—semantic alignment followed by joint semantic and paralinguistic alignment—demonstrating that self-generated supervision can yield robust multimodal alignment with limited data. The results suggest a scalable, instruction-efficient path for multimodal LLMs and motivate extending the SIFT paradigm to other modalities such as vision.

Abstract

Extending large language models (LLMs) to the speech domain has recently gained significant attention. A typical approach connects a pretrained LLM with an audio encoder through a projection module and trains the resulting model on large-scale, task-specific instruction-tuning datasets. However, curating such instruction-tuning data for specific requirements is time-consuming, and models trained in this manner often generalize poorly to unseen tasks. In this work, we first formulate that the strongest generalization of a speech-LLM is achieved when it is trained with Self-Generated Instruction-Free Tuning (SIFT), in which supervision signals are generated by a frozen LLM using textual representations of speech as input. Our proposed SIFT paradigm eliminates the need for collecting task-specific question-answer pairs and yields the theoretically best generalization to unseen tasks. Building upon this paradigm, we introduce AZeroS (Auden Zero-instruction-tuned Speech-LLM), which is trained on speech-text pairs derived from publicly available corpora, including approximately 25,000 hours of speech with ASR transcripts and 3,000 hours of speech with paralinguistic labels. Built upon Qwen2.5-7B-Instruct, the model updates only two lightweight projection modules (23.8 million parameters each), while keeping both the LLM and audio encoders frozen. Despite the minimal training cost and modest data scale, AZeroS achieves state-of-the-art performance on both semantic and paralinguistic benchmarks, including VoiceBench, AIR-Bench Foundation (Speech), and AIR-Bench Chat (Speech).

Figures (5)

• Figure 1: Overview of the proposed AZeroS framework built upon a frozen Qwen2.5-7B-Instruct backbone. The system couples a Self-Generated Data Construction pipeline (left) with a two-stage Self-Generated Instruction-Free Tuning (SIFT) training procedure (right). In Stage 1 (Semantic), the LLM generates targets from speech transcripts to train a projector on the semantic TTAauden_tta encoder. In Stage 2 (Semantic + Paralinguistic), we augment inputs with metadata (e.g., emotion, gender, age); the resulting rich responses train a second projector on a paralinguistic Auden-Voiceauden_voice encoder. Together, this progressive SIFT paradigm extends the model's capability from purely semantic to joint semantic–paralinguistic understanding without relying on task-specific instructions.
• Figure 2: Illustration of semantic-only instruction-tuning configurations: (a) TSITs, (b) SITs, and (c) SIFTs. Examples are selected from an utterance in the IEMOCAP dataset busso2008iemocap, with the transcript "what shall we do if they suddenly walk in on us". ([...] indicates truncated text. Detailed examples are in Appendix \ref{['app:example']}.)
• Figure 3: Illustration of semantic and paralinguistic tuning configurations: (a) SITsp and (b) SIFTsp. Examples are selected from an utterance in the IEMOCAP dataset busso2008iemocap, with the transcript "what shall we do if they suddenly walk in on us" and paralinguistic metadata (gender: female, emotion: happy). ([...] indicates truncated text. Detailed examples are in Appendix \ref{['app:example']}.)
• Figure 4: Comparison of tuning configurations for the Dialog-LLM: (a) SITssp and (b) SIFTssp. By injecting a system prompt to enforce natural conversation, the LLM's default behavior changes: it tends to prioritize conciseness over detail, leading to the loss of paralinguistic information in the SIFT targets (as seen in (b)). Explicit instructions (SIT) are required to force the model to verbalize these details (a). ([...] indicates truncated text. Detailed examples are in Appendix \ref{['app:example']}.)
• Figure 5: Comparison of intrinsic response behaviors between the two alignment targets at inference time (without system prompts). The ssp model (First Replier), trained with SIFT+SIT, acts as a Speech Dialog Role, providing natural and concise human-like responses. In contrast, the sp model (Second Replier), trained with SIFT, acts as a Speech Understanding Role, providing detailed analytical descriptions.