Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal In-context Learning for ASR of Low-resource Languages

Zhaolin Li, Jan Niehues

TL;DR

This work investigates multimodal in-context learning (MICL) for automatic speech recognition (ASR) of low-resource languages by leveraging speech LLMs Phi-4 and Qwen3-Omni across three endangered languages. It formalizes MICL Task Description, prompts, sample selection with SONAR, cross-lingual instruction fine-tuning, and a MICL-based hypothesis re-ranking system that combines acoustic scores with language-model guidance. Key findings show MICL enables learning unseen languages from audio–text demonstrations, cross-lingual fine-tuning improves generalization without target-language data, and attention analyses reveal layer-dependent, text-biased modality engagement; MICL-based hypothesis selection enhances ASR performance over purely acoustic baselines. The results suggest practical pathways for robust ASR in underrepresented languages and highlight the value of multilingual context and multimodal reasoning in speech LLMs.

Abstract

Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context learning (ICL) with large language models (LLMs) addresses this problem, but prior work largely focuses on high-resource languages covered during training and text-only settings. This paper investigates whether speech LLMs can learn unseen languages with multimodal ICL (MICL), and how this learning can be used to improve ASR. We conduct experiments with two speech LLMs, Phi-4 and Qwen3-Omni, on three diverse endangered languages. Firstly, we find that MICL is effective for unseen languages, leveraging both speech and text modalities. We further show that cross-lingual transfer learning improves MICL efficiency on target languages without training on them. Moreover, we analyze attention patterns to interpret MICL mechanisms, and we observe layer-dependent preferences between audio and text context, with an overall bias towards text. Finally, we show that prompt-based ASR with speech LLMs performs poorly on unseen languages, motivating a simple ASR system that combines a stronger acoustic model with a speech LLM via MICL-based selection of acoustic hypotheses. Results show that MICL consistently improves ASR performance, and that cross-lingual transfer learning matches or outperforms corpus-trained language models without using target-language data. Our code is publicly available.

Multimodal In-context Learning for ASR of Low-resource Languages

TL;DR

This work investigates multimodal in-context learning (MICL) for automatic speech recognition (ASR) of low-resource languages by leveraging speech LLMs Phi-4 and Qwen3-Omni across three endangered languages. It formalizes MICL Task Description, prompts, sample selection with SONAR, cross-lingual instruction fine-tuning, and a MICL-based hypothesis re-ranking system that combines acoustic scores with language-model guidance. Key findings show MICL enables learning unseen languages from audio–text demonstrations, cross-lingual fine-tuning improves generalization without target-language data, and attention analyses reveal layer-dependent, text-biased modality engagement; MICL-based hypothesis selection enhances ASR performance over purely acoustic baselines. The results suggest practical pathways for robust ASR in underrepresented languages and highlight the value of multilingual context and multimodal reasoning in speech LLMs.

Abstract

Automatic speech recognition (ASR) still covers only a small fraction of the world's languages, mainly due to supervised data scarcity. In-context learning (ICL) with large language models (LLMs) addresses this problem, but prior work largely focuses on high-resource languages covered during training and text-only settings. This paper investigates whether speech LLMs can learn unseen languages with multimodal ICL (MICL), and how this learning can be used to improve ASR. We conduct experiments with two speech LLMs, Phi-4 and Qwen3-Omni, on three diverse endangered languages. Firstly, we find that MICL is effective for unseen languages, leveraging both speech and text modalities. We further show that cross-lingual transfer learning improves MICL efficiency on target languages without training on them. Moreover, we analyze attention patterns to interpret MICL mechanisms, and we observe layer-dependent preferences between audio and text context, with an overall bias towards text. Finally, we show that prompt-based ASR with speech LLMs performs poorly on unseen languages, motivating a simple ASR system that combines a stronger acoustic model with a speech LLM via MICL-based selection of acoustic hypotheses. Results show that MICL consistently improves ASR performance, and that cross-lingual transfer learning matches or outperforms corpus-trained language models without using target-language data. Our code is publicly available.
Paper Structure (32 sections, 4 equations, 2 figures, 15 tables)

This paper contains 32 sections, 4 equations, 2 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Task Description
  4. Prompt Modality Design
  5. Sample Selection Strategy
  6. Cross-lingual Instruction Fine-tuning
  7. ASR Hypothesis Selection System
  8. Experimental setup
  9. Datasets
  10. Models
  11. Fine-tuning
  12. Hypotheses Generation
  13. Evaluation Metrics
  14. Results and Analysis
  15. Speech LLMs perform MICL
  16. ...and 17 more sections

Figures (2)

  • Figure 1: Layer-wise attention allocation across demonstrations of 5 and 10 samples using Phi-4, averaged over all attention heads on the Khinalug dataset.
  • Figure 2: Layer-wise attention allocation under different intervention settings. We show attention allocation with no modification (left 1), with gold text replacing sample 4 (left 2), with gold audio replacing sample 4 (right 2), and with both gold text and audio replacing sample 4 (right 1). We include the attentions on target audio for reference purposes.