Listening Between the Frames: Bridging Temporal Gaps in Large Audio-Language Models

TL;DR

This work tackles the challenge of enabling large audio-language models to perform fine-grained temporal reasoning on long audio. It introduces TimeAudio, a time-sensitive LALM that uses temporal markers, absolute time-aware encoding, and segment-level token merging, trained in a two-stage pipeline and supported by the FTAR dataset. Across dense captioning, temporal grounding, and timeline summarization, TimeAudio delivers strong temporal localization and grounding accuracy, surpassing several baselines and existing LALMs. The approach demonstrates that explicit temporal grounding and efficient long-audio processing are crucial for robust, time-aware audio understanding with LLMs.

Abstract

Recent Large Audio-Language Models (LALMs) exhibit impressive capabilities in understanding audio content for conversational QA tasks. However, these models struggle to accurately understand timestamps for temporal localization (e.g., Temporal Audio Grounding) and are restricted to short audio perception, leading to constrained capabilities on fine-grained tasks. We identify three key aspects that limit their temporal localization and long audio understanding: (i) timestamp representation, (ii) architecture, and (iii) data. To address this, we introduce TimeAudio, a novel method that empowers LALMs to connect their understanding of audio content with precise temporal perception. Specifically, we incorporate unique temporal markers to improve time-sensitive reasoning and apply an absolute time-aware encoding that explicitly grounds the acoustic features with absolute time information. Moreover, to achieve end-to-end long audio understanding, we introduce a segment-level token merging module to substantially reduce audio token redundancy and enhance the efficiency of information extraction. Due to the lack of suitable datasets and evaluation metrics, we consolidate existing audio datasets into a new dataset focused on temporal tasks and establish a series of metrics to evaluate the fine-grained performance. Evaluations show strong performance across a variety of fine-grained tasks, such as dense captioning, temporal grounding, and timeline speech summarization, demonstrating TimeAudio's robust temporal localization and reasoning capabilities.

Figures (7)

• Figure 1: Example of failed cases by Qwen2-Audio and Qwen2-Audio-R1 on fine-grained tasks that require both semantics and timestamps as output.
• Figure 2: Overview of our TimeAudio method. The input audio is first split into segments and encoded into audio tokens. The window Q-former then projects these audio tokens into the language space and utilize a segment-level token merging to retain important semantic information along time. Timestamps are converted to special anchor and offset tokens.
• Figure 3: Involved tasks and datasets in the time-aware instruction tuning dataset.
• Figure 4: Qualitative results between models on the dense audio captioning task and timeline speech summarization task. Yellow denotes the time interval and red marks the audio content.
• Figure 5: Qualitative results on dense audio captioning task. For each audio, we ask audio to detect a series of events in the given audio and output the corresponding timestamps and descriptions.