Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Extending Audio Context for Long-Form Understanding in Large Audio-Language Models

Yuatyong Chaichana, Pittawat Taveekitworachai, Warit Sirichotedumrong, Potsawee Manakul, Kunat Pipatanakul

TL;DR

This work tackles the bottleneck of short audio context in Large Audio-Language Models by introducing Partial YaRN, an audio-only RoPE extension that preserves text-position encodings, and VLAT, a training strategy to generalize to long-form audio. Partial YaRN extends the audio window through targeted RoPE manipulation, while VLAT simulates diverse long contexts during fine-tuning to improve robustness for unseen lengths. Training-free extensions yield model- and length-dependent gains, with notable improvements when anchoring extensions to observed 2-minute contexts. VLAT further enhances long-form generalization and pairs well with inference-time extensions, offering a data-efficient path to robust long-form audio understanding in LALMs.

Abstract

Large Audio-Language Models (LALMs) are often constrained by short audio context windows, even when their text backbones support long contexts, limiting long-form audio understanding. Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. First, building on RoPE-based context extension, we introduce Partial YaRN, a training-free, audio-only extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM's text capabilities. Second, we propose Virtual Longform Audio Training (VLAT), a training strategy that extends Partial YaRN into a training-time positional augmentation. VLAT simulates diverse audio lengths during training, enabling generalization to inputs far longer than those seen in training and improving robustness for long-context audio understanding. Our experiments on SALMONN and Qwen2-Audio show that Partial YaRN outperforms the original models across wide range of settings, and VLAT training strategy provides substantial improvement, achieving strong performance on long audio of unseen lengths.

Extending Audio Context for Long-Form Understanding in Large Audio-Language Models

TL;DR

This work tackles the bottleneck of short audio context in Large Audio-Language Models by introducing Partial YaRN, an audio-only RoPE extension that preserves text-position encodings, and VLAT, a training strategy to generalize to long-form audio. Partial YaRN extends the audio window through targeted RoPE manipulation, while VLAT simulates diverse long contexts during fine-tuning to improve robustness for unseen lengths. Training-free extensions yield model- and length-dependent gains, with notable improvements when anchoring extensions to observed 2-minute contexts. VLAT further enhances long-form generalization and pairs well with inference-time extensions, offering a data-efficient path to robust long-form audio understanding in LALMs.

Abstract

Large Audio-Language Models (LALMs) are often constrained by short audio context windows, even when their text backbones support long contexts, limiting long-form audio understanding. Prior work has introduced context-extension methods (e.g. YaRN) on unimodal LLMs, yet their application to LALMs remains unexplored. First, building on RoPE-based context extension, we introduce Partial YaRN, a training-free, audio-only extension method that modifies only audio token positions, leaving text positions intact to preserve the base LLM's text capabilities. Second, we propose Virtual Longform Audio Training (VLAT), a training strategy that extends Partial YaRN into a training-time positional augmentation. VLAT simulates diverse audio lengths during training, enabling generalization to inputs far longer than those seen in training and improving robustness for long-context audio understanding. Our experiments on SALMONN and Qwen2-Audio show that Partial YaRN outperforms the original models across wide range of settings, and VLAT training strategy provides substantial improvement, achieving strong performance on long audio of unseen lengths.
Paper Structure (58 sections, 6 equations, 7 figures, 6 tables)

This paper contains 58 sections, 6 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Rotary Positional Encoding
  4. Context Length Extension
  5. Position interpolation (PI).
  6. YaRN.
  7. Other methods.
  8. Large Audio-language Models and Their Audio Contexts
  9. Partial YaRN
  10. Methodology
  11. Hyperparameters
  12. Experimental Setup
  13. Data
  14. YODAS2-MCQA
  15. Models
  16. ...and 43 more sections

Figures (7)

  • Figure 1: An illustration of Partial YaRN. The top plot displays low (blue) and high (red) frequency dimensions of RoPE. The bottom plot displays RoPE with Partial YaRN applied. In this example, the audio window of the low frequency dimensions are interpolated (stretched) by a factor of 1.75x to accommodate longer audio inputs, while the high frequency dimensions remain unaltered to preserve local distances and high frequency information yarn. Within the Extended Audio Region, the magnitudes of every dimension is scaled to serve as a proxy for attention temperature scaling. Despite that the scaling can create a minor discontinuity in the signal, information from other dimensions can help the model disambiguate this.
  • Figure 2: Prompt used for inferencing on YODAS2-MCQA.
  • Figure 3: Hyperparameter Sensitivity of Partial YaRN on Qwen2-Audio (extending the original 30s audio context).
  • Figure 4: Hyperparameter Sensitivity of Partial YaRN on Qwen2-Audio (extending the observed 2mins audio context).
  • Figure 5: Hyperparameter Sensitivity of Partial YaRN on SALMONN (extending the original 30s audio context).
  • ...and 2 more figures