Spoken Language Understanding on Unseen Tasks With In-Context Learning
Neeraj Agrawal, Sriram Ganapathy
TL;DR
This paper tackles the challenge of applying spoken language understanding (SLU) to unseen tasks when task-specific labeled data is scarce. It shows that zero-/few-shot performance of speech–text LLMs is insufficient and proposes robust task-agnostic fine-tuning, notably randomized label fine-tuning, to improve generalization across unseen SLU tasks. Through experiments on three SLUE datasets, the authors demonstrate that randomized label fine-tuning often yields the strongest performance in mis-matched train/evaluation settings, surpassing regular fine-tuning and no fine-tuning, while avoiding task-specific annotations. The work advances cross-task and cross-modal generalization for SLU using a unified prompting setup with SALMONN and LoRA, highlighting practical implications for deploying SLU systems with limited labeled data.
Abstract
Spoken language understanding (SLU) tasks involve diverse skills that probe the information extraction, classification and/or generation capabilities of models. In this setting, task-specific training data may not always be available. While traditional task-specific SLU models are unable to cater to such requirements, the speech-text large language models (LLMs) offer a promising alternative with emergent abilities. However, out of-the-box, our evaluations indicate that the zero/few-shot performance of prominent open-source speech-text LLMs on SLU tasks are not up to the mark. In this paper, we introduce a novel approach to robust task-agnostic fine-tuning using randomized class labels. With this proposed fine-tuning, we illustrate that the performance of the speech-text LLMs on an unseen task is significantly improved over standard approaches. Critically, the proposed approach avoids the requirement of task-specific data annotations for enabling new tasks in speech-text LLMs.