AHA: Aligning Large Audio-Language Models for Reasoning Hallucinations via Counterfactual Hard Negatives

TL;DR

This work tackles the persistent issue of hallucinations in large audio-language models by introducing Audio Hallucination Alignment (AHA), a post-training framework that uses counterfactual hard negatives to enforce strict acoustic grounding across temporal reasoning tasks. AHA builds a dual dataset pipeline (alignment and evaluation) around a shared Audio-Question pool and introduces AHA-Eval, a diagnostic benchmark to quantify fine-grained temporal reasoning failures. Applying AHA with Direct Preference Optimization and LoRA to Qwen2.5-Omni yields Qwen-Audio-AHA, which significantly reduces four targeted hallucination types and improves performance on both diagnostic and public benchmarks, indicating strong generalization. The study highlights that aligning temporal grounding not only reduces hallucinations but also enhances general reasoning, while identifying limitations in current automated evaluation methods that can misjudge semantic equivalence and reasoning depth. Overall, AHA provides a targeted, model-agnostic approach to strengthen grounding in LALMs and offers a rigorous framework for future evaluation of audio-grounded reasoning systems.

Abstract

Although Large Audio-Language Models (LALMs) deliver state-of-the-art (SOTA) performance, they frequently suffer from hallucinations, e.g. generating text not grounded in the audio input. We analyze these grounding failures and identify a distinct taxonomy: Event Omission, False Event Identity, Temporal Relation Error, and Quantitative Temporal Error. To address this, we introduce the AHA (Audio Hallucination Alignment) framework. By leveraging counterfactual hard negative mining, our pipeline constructs a high-quality preference dataset that forces models to distinguish strict acoustic evidence from linguistically plausible fabrications. Additionally, we establish AHA-Eval, a diagnostic benchmark designed to rigorously test these fine-grained temporal reasoning capabilities. We apply this data to align Qwen2.5-Omni. The resulting model, Qwen-Audio-AHA, achieves a 13.7% improvement on AHA-Eval. Crucially, this benefit generalizes beyond our diagnostic set. Our model shows substantial gains on public benchmarks, including 1.3% on MMAU-Test and 1.6% on MMAR, outperforming latest SOTA methods.

Figures (7)

• Figure 1: Illustrative examples showcasing the improvements after performing the proposed AHA framework alignment. Compared to the base model Qwen2.5-Omni, our Qwen-Audio-AHA effectively mitigates hallucinations and errors across four critical dimensions: (1) Event Omission, (2) False Event Identity, (3) Temporal Relation Error, and (4) Quantitative Temporal Error.
• Figure 2: Four principal hallucination types observed in temporal audio reasoning: Event Omission, False Event Identity, Temporal Relation Error, and Quantitative Temporal Error. Each panel illustrates the definition of the failure mode along with a representative hallucination example produced by Qwen2.5-Omni on diagnostic testing.
• Figure 3: The unified data construction pipeline for AHA. The process begins by establishing a shared Audio-Question Pool ($\mathcal{D}$) derived from complex acoustic scenes and hallucination-sensitive fine-grained reasoning templates. From this foundation, the pipeline bifurcates into two complementary views: (Left) The Alignment View constructs preference pairs for DPO by contrasting caption-derived chosen responses ($r_i^+$) against LLM-generated rejected responses ($r_i^-$) that contain specific hallucination patterns. (Right) The Evaluation View establishes a rigorous QA benchmark by collecting human-verified ground-truth ($y_i^*$) and annotating fine-grained hallucination types ($\tau(q_i)$).
• Figure 4: Categorical accuracies before and after alignment in MMAU-test benchmark. Most dimensions have improved accuracy after alignment, especially Temporal Event Reasoning (TER) and Phonological Sequence Decoding (PSD), while stagnation or minor degradation are observed in a few subcategories.
• Figure 5: A critical failure case of automated LLM evaluation. The LALM correctly identifies the acoustic event as "people laughing", which is semantically equivalent to the caption's "chuckle or chortle". However, the LLM judge fails to bridge this semantic gap, incorrectly penalizing the model with Event Omission and False Event Identity errors. This highlights the risk of false positives in current hallucination benchmarks.