SpeechLLMs for Large-scale Contextualized Zero-shot Slot Filling

TL;DR

The paper investigates SpeechLLMs for large-scale zero-shot slot filling by unifying speech and text processing in a decoder-only architecture comprising a speech encoder, modality adapter, and LLM. It builds a large CallCenter-A dataset annotated with GPT-4o, creates an instruction-based training regime, and augments training with multitask data to improve robustness and generalization. Through systematic exploration of adapter types, LLM fine-tuning strategies, and multistage training, the authors demonstrate substantial gains toward a text-based oracle while revealing persisting gaps in out-of-domain and zero-shot performance. The work offers practical guidance on architecture choices, training strategies, and data expansion for scalable, efficient multimodal SLU systems, and discusses tradeoffs between composite and foundational models for real-world deployment.

Abstract

Slot filling is a crucial subtask in spoken language understanding (SLU), traditionally implemented as a cascade of speech recognition followed by one or more natural language understanding (NLU) components. The recent advent of speech-based large language models (speechLLMs), which integrate speech and textual foundation models, has opened new avenues for achieving speech understanding tasks in a more unified, generative, and instruction-following manner while promising data and compute efficiency with zero-shot abilities, generalizing to unseen slot labels. We address the slot-filling task by creating an empirical upper bound for the task, identifying performance, robustness, and generalization gaps, and proposing improvements to the training data, architecture, and training strategies to narrow the gap with the upper bound result. We show that each of these measures improve performance substantially, while highlighting practical challenges and providing empirical guidance and insights for harnessing these emerging models.

Figures (5)

• Figure 1: SpeechLLM architecture
• Figure 2: Learning curves showing training difficulties
• Figure 3: Learning curves comparing multistage training against single-stage joint fine-tuning.
• Figure 4: The prompt used for GPT-4o, [System]+[User] and its response/completion as [Assistant] which is the desired JSON annotation for turns in the dialog transcript. The $\{$dialog$\_$transcript$\}$ is the complete human transcription of the conversation. All the named entities mentioned in the examples are fictitious.
• Figure 5: A subset of prompts used for the "instruction" part of the training triplet $\{$audio, instruction, response$\}$. The fields "slots" and "context" in each prompt are populated as described in Section 2.2.