Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SICL-AT: Another way to adapt Auditory LLM to low-resource task

Haolong Zheng, Siyin Wang, Zengrui Jin, Mark Hasegawa-Johnson

TL;DR

This paper tackles the brittleness of fine-tuning for low-resource audio tasks by leveraging in-context learning (ICL) and introducing Speech In-Context Learning Adaptation-Tuning (SICL-AT), a post-training recipe that trains auditory LLMs to utilize audio demonstrations at inference. Vanilla ICL already improves zero-shot performance across ASR, audio understanding/reasoning, and multilingual tasks, motivating a gradient-free, demonstration-conditioned adaptation approach. SICL-AT further strengthens ICL by training on high-resource, diverse speech data in an episodic format, yielding robust gains across ASR, ST, and AU/AR and outperforming direct fine-tuning under distribution shift. The findings highlight the practical utility of demonstration-conditioned inference for translating abundant high-resource data into improved performance on scarce, low-resource speech tasks, with the alignment of post-training data to downstream formats proving particularly beneficial.

Abstract

Auditory Large Language Models (LLMs) have demonstrated strong performance across a wide range of speech and audio understanding tasks. Nevertheless, they often struggle when applied to low-resource or unfamiliar tasks. In case of labeled in-domain data is scarce or mismatched to the true test distribution, direct fine-tuning can be brittle. In-Context Learning (ICL) provides a training-free, inference-time solution by adapting auditory LLMs through conditioning on a few in-domain demonstrations. In this work, we first show that \emph{Vanilla ICL}, improves zero-shot performance across diverse speech and audio tasks for selected models which suggest this ICL adaptation capability can be generalized to multimodal setting. Building on this, we propose \textbf{Speech In-Context Learning Adaptation Training (SICL-AT)}, a post-training recipe utilizes only high resource speech data intending to strengthen model's in-context learning capability. The enhancement can generalize to audio understanding/reasoning task. Experiments indicate our proposed method consistently outperforms direct fine-tuning in low-resource scenario.

SICL-AT: Another way to adapt Auditory LLM to low-resource task

TL;DR

This paper tackles the brittleness of fine-tuning for low-resource audio tasks by leveraging in-context learning (ICL) and introducing Speech In-Context Learning Adaptation-Tuning (SICL-AT), a post-training recipe that trains auditory LLMs to utilize audio demonstrations at inference. Vanilla ICL already improves zero-shot performance across ASR, audio understanding/reasoning, and multilingual tasks, motivating a gradient-free, demonstration-conditioned adaptation approach. SICL-AT further strengthens ICL by training on high-resource, diverse speech data in an episodic format, yielding robust gains across ASR, ST, and AU/AR and outperforming direct fine-tuning under distribution shift. The findings highlight the practical utility of demonstration-conditioned inference for translating abundant high-resource data into improved performance on scarce, low-resource speech tasks, with the alignment of post-training data to downstream formats proving particularly beneficial.

Abstract

Auditory Large Language Models (LLMs) have demonstrated strong performance across a wide range of speech and audio understanding tasks. Nevertheless, they often struggle when applied to low-resource or unfamiliar tasks. In case of labeled in-domain data is scarce or mismatched to the true test distribution, direct fine-tuning can be brittle. In-Context Learning (ICL) provides a training-free, inference-time solution by adapting auditory LLMs through conditioning on a few in-domain demonstrations. In this work, we first show that \emph{Vanilla ICL}, improves zero-shot performance across diverse speech and audio tasks for selected models which suggest this ICL adaptation capability can be generalized to multimodal setting. Building on this, we propose \textbf{Speech In-Context Learning Adaptation Training (SICL-AT)}, a post-training recipe utilizes only high resource speech data intending to strengthen model's in-context learning capability. The enhancement can generalize to audio understanding/reasoning task. Experiments indicate our proposed method consistently outperforms direct fine-tuning in low-resource scenario.
Paper Structure (19 sections, 1 figure, 6 tables, 1 algorithm)

This paper contains 19 sections, 1 figure, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Training Instance
  4. Training Data
  5. Models
  6. Evaluation Data & Metrics
  7. Child's ASR.
  8. Audio Understanding/Reasoning.
  9. Multilingual ASR & Speech Translation.
  10. Experiments
  11. Vanilla In-Context Learning
  12. Effect of SICL Fine-tuning
  13. SICL-AT1 (ASR-only).
  14. SICL-AT2 (ASR + ST).
  15. SICL-AT3 (ASR + ST + SQA).
  16. ...and 4 more sections

Figures (1)

  • Figure 1: Motivation and overview of SICL-AT for low-resource audio tasks. Left: Direct supervised fine-tuning (SFT) on limited in-domain data often degrades performance under distribution shift due to domain mismatch between scarce training samples and the target test distribution. Right: We instead perform SICL-style fine-tuning on abundant, high-resource speech tasks using an ICL formatted objective, then apply SICL-style inference by providing task demonstrations at test time to adapt the same LoRA-augmented auditory LLM to low-resource domains, improving robustness and downstream performance.