Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data

Chun-Yi Kuan, Hung-yi Lee

TL;DR

BALSa introduces a backbone-LLM driven synthetic data framework to bootstrap audio-language alignment for ALLMs, producing contrastive (present vs absent) and multi-audio data without altering the backbone. By freezing the audio encoder and LLM while training a lightweight modality adapter, BALSa achieves data-efficient training and competitive performance across audio QA, reasoning, and safety benchmarks, while effectively mitigating audio hallucinations. BALSa-MA extends to multi-audio scenarios, enabling difference comparison and joint captioning that further boosts understanding and reasoning, especially with stronger backbones. The approach demonstrates robustness across backbones, data prompts, and fine-tuning strategies, offering a practical, scalable path for reliable, instruction-following ALLMs with reduced data and compute requirements.

Abstract

Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs. These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks. This adaptation process presents two major limitations. First, ALLMs often suffer from catastrophic forgetting, where crucial textual capabilities like instruction-following are lost after training on audio data. In some cases, models may even hallucinate sounds that are not present in the input audio, raising concerns about reliability. Second, achieving cross-modal alignment between audio and language typically relies on large collections of task-specific question-answer pairs for instruction tuning, making it resource-intensive. To address these issues, previous works have leveraged the backbone LLMs to synthesize general-purpose, caption-style alignment data. In this paper, we propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds. We further extend our approach to multi-audio scenarios, enabling the model to either explain differences between audio inputs or produce unified captions that describe all inputs, thereby enhancing audio-language alignment. We refer to the entire ALLM training framework as bootstrapping audio-language alignment via synthetic data generation from backbone LLMs (BALSa). Experimental results indicate that our method effectively mitigates audio hallucinations while reliably maintaining strong performance on audio understanding and reasoning benchmarks, as well as instruction-following skills. Moreover, incorporating multi-audio training further enhances the model's comprehension and reasoning capabilities. Overall, BALSa offers an efficient and scalable approach to developing ALLMs.

From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data

TL;DR

BALSa introduces a backbone-LLM driven synthetic data framework to bootstrap audio-language alignment for ALLMs, producing contrastive (present vs absent) and multi-audio data without altering the backbone. By freezing the audio encoder and LLM while training a lightweight modality adapter, BALSa achieves data-efficient training and competitive performance across audio QA, reasoning, and safety benchmarks, while effectively mitigating audio hallucinations. BALSa-MA extends to multi-audio scenarios, enabling difference comparison and joint captioning that further boosts understanding and reasoning, especially with stronger backbones. The approach demonstrates robustness across backbones, data prompts, and fine-tuning strategies, offering a practical, scalable path for reliable, instruction-following ALLMs with reduced data and compute requirements.

Abstract

Audio-aware large language models (ALLMs) have recently made great strides in understanding and processing audio inputs. These models are typically adapted from text-based large language models (LLMs) through additional training on audio-related tasks. This adaptation process presents two major limitations. First, ALLMs often suffer from catastrophic forgetting, where crucial textual capabilities like instruction-following are lost after training on audio data. In some cases, models may even hallucinate sounds that are not present in the input audio, raising concerns about reliability. Second, achieving cross-modal alignment between audio and language typically relies on large collections of task-specific question-answer pairs for instruction tuning, making it resource-intensive. To address these issues, previous works have leveraged the backbone LLMs to synthesize general-purpose, caption-style alignment data. In this paper, we propose a data generation framework that produces contrastive-like training data, designed to enhance ALLMs' ability to differentiate between present and absent sounds. We further extend our approach to multi-audio scenarios, enabling the model to either explain differences between audio inputs or produce unified captions that describe all inputs, thereby enhancing audio-language alignment. We refer to the entire ALLM training framework as bootstrapping audio-language alignment via synthetic data generation from backbone LLMs (BALSa). Experimental results indicate that our method effectively mitigates audio hallucinations while reliably maintaining strong performance on audio understanding and reasoning benchmarks, as well as instruction-following skills. Moreover, incorporating multi-audio training further enhances the model's comprehension and reasoning capabilities. Overall, BALSa offers an efficient and scalable approach to developing ALLMs.

Paper Structure

This paper contains 24 sections, 3 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Bootstrapping Audio-Language Alignment via Synthetic Data Generation from Backbone LLMs
  5. BALSa-MA: Bootstrapping Multi-Audio Learning via Difference Comparison and Joint Captioning
  6. Model Design and Training Approach
  7. Experimental Setup
  8. Training Datasets
  9. Model Selection and Training Setups
  10. Baseline Models and Evaluation Setups
  11. Ablation Study
  12. Evaluation Benchmarks
  13. Audio Question Answering Benchmark
  14. Audio Reasoning Benchmark
  15. Reliability and Safety Benchmark
  16. ...and 9 more sections

Figures (3)

  • Figure 1: Benchmark performance of models trained on captioning data generated with five distinct prompts. The combined-prompt setting (red dashed line) consistently yields the strongest results and reduces sensitivity to individual prompt formulations.
  • Figure 2: Performance of BALSa across varying training set sizes. Results show consistent improvements as data increases, while maintaining stable performance even with limited training data.
  • Figure 3: Comparison of average performance across benchmarks and training data (log scale). Circle size indicates the amount of training data.