Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions

TL;DR

The paper tackles transcription in multi-talker settings by introducing MT-LLM, a framework that pairs dual speech encoders with a fine-tuned LLM to follow versatile transcription instructions. It treats ASR as autoregressive next-token prediction conditioned on input speech and an instruction, formalized as $p(\mathbf{Y} \mid \mathbf{X}, \mathbf{I}; \boldsymbol{\theta}) = \prod_{t=0}^{T-1} p(y_t \mid \mathbf{X}, \mathbf{I}, \mathbf{Y}_{<t}; \boldsymbol{\theta})$, and uses Serialized Output Training to resolve speaker permutation. The architecture employs Whisper and WavLM encoders, adapters, LoRA fine-tuning on Llama-2-chat, and a total of about $7.55\times 10^9$ parameters with $1\%$ trainable, achieving instruction-based MT ASR across six task variations (MT, TT, KT, SS, OS, TL) on simulated LibriSpeechMix and CoVoST German data. Experimental results show competitive performance against specialized multi-talker models, with notable strengths in instruction following and single-talker generalization, and ablations highlighting the usefulness of dual encoders and multitask training. The work points to a scalable direction for natural human–computer interaction in complex speech environments using parameter-efficient learning and accessible code.

Abstract

Recent advancements in large language models (LLMs) have revolutionized various domains, bringing significant progress and new opportunities. Despite progress in speech-related tasks, LLMs have not been sufficiently explored in multi-talker scenarios. In this work, we present a pioneering effort to investigate the capability of LLMs in transcribing speech in multi-talker environments, following versatile instructions related to multi-talker automatic speech recognition (ASR), target talker ASR, and ASR based on specific talker attributes such as sex, occurrence order, language, and keyword spoken. Our approach utilizes WavLM and Whisper encoder to extract multi-faceted speech representations that are sensitive to speaker characteristics and semantic context. These representations are then fed into an LLM fine-tuned using LoRA, enabling the capabilities for speech comprehension and transcription. Comprehensive experiments reveal the promising performance of our proposed system, MT-LLM, in cocktail party scenarios, highlighting the potential of LLM to handle speech-related tasks based on user instructions in such complex settings. The code, model, and samples are available at https://github.com/cuhealthybrains/MT-LLM.

Figures (2)

• Figure 1: MT-LLM supports versatile ASR instructions in multi-talker scenarios. Given a multi-talker overlapped speech input (a) and a text instruction prompt (b), the proposed MT-LLM (c) is expected to autoregressively generate corresponding target transcripts (d). For tasks that involve multiple talkers, MT-LLM follows the SOT-style output, transcribing the utterances of multiple talkers in the order of their start times, separated by “$<$sc$>$” indicating “speaker change”.
• Figure 2: Model architecture of MT-LLM. Multi-faceted speech representations are extracted using dual speech encoders and projected into the LLM feature space. Fine-tuned with LoRA, the LLM acquires the capability to comprehend and transcribe speech in multi-talker scenarios based on text instructions.