Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Frame-Level Internal Tool Use for Temporal Grounding in Audio LMs

Joesph An, Phillip Keung, Jiaqi Wang, Orevaoghene Ahia, Noah A. Smith

TL;DR

The paper addresses the challenge of precise temporal grounding in audio language models, where generating timestamps as text tokens leads to hallucinations and poor length generalization. It introduces frame-level internal tool use, attaching a lightweight prediction head to the model's internal audio representations to produce frame-level event distributions, and trains with either a binary frame-level loss or an inhomogeneous Poisson process (IHP) loss. Across word localization, speaker diarization, and audio event localization, Poisson-based frame-level training consistently outperforms token-based generation and binary loss baselines, delivering high accuracy and robustness to out-of-distribution durations, while enabling substantial inference speedups (often exceeding $50\times$, up to $60\times$). The approach leverages time-rescaling and posterior-mode inference for continuous-time event modeling and demonstrates strong generalization to multiple timestamps and longer sequences, representing a practical, efficient alternative to autoregressive timestamp generation in audio LMs.

Abstract

Large audio language models are increasingly used for complex audio understanding tasks, but they struggle with temporal tasks that require precise temporal grounding, such as word alignment and speaker diarization. The standard approach, where we generate timestamps as sequences of text tokens, is computationally expensive and prone to hallucination, especially when processing audio lengths outside the model's training distribution. In this work, we propose frame-level internal tool use, a method that trains audio LMs to use their own internal audio representations to perform temporal grounding directly. We introduce a lightweight prediction mechanism trained via two objectives: a binary frame classifier and a novel inhomogeneous Poisson process (IHP) loss that models temporal event intensity. Across word localization, speaker diarization, and event localization tasks, our approach outperforms token-based baselines. Most notably, it achieves a >50x inference speedup and demonstrates robust length generalization, maintaining high accuracy on out-of-distribution audio durations where standard token-based models collapse completely.

Frame-Level Internal Tool Use for Temporal Grounding in Audio LMs

TL;DR

The paper addresses the challenge of precise temporal grounding in audio language models, where generating timestamps as text tokens leads to hallucinations and poor length generalization. It introduces frame-level internal tool use, attaching a lightweight prediction head to the model's internal audio representations to produce frame-level event distributions, and trains with either a binary frame-level loss or an inhomogeneous Poisson process (IHP) loss. Across word localization, speaker diarization, and audio event localization, Poisson-based frame-level training consistently outperforms token-based generation and binary loss baselines, delivering high accuracy and robustness to out-of-distribution durations, while enabling substantial inference speedups (often exceeding , up to ). The approach leverages time-rescaling and posterior-mode inference for continuous-time event modeling and demonstrates strong generalization to multiple timestamps and longer sequences, representing a practical, efficient alternative to autoregressive timestamp generation in audio LMs.

Abstract

Large audio language models are increasingly used for complex audio understanding tasks, but they struggle with temporal tasks that require precise temporal grounding, such as word alignment and speaker diarization. The standard approach, where we generate timestamps as sequences of text tokens, is computationally expensive and prone to hallucination, especially when processing audio lengths outside the model's training distribution. In this work, we propose frame-level internal tool use, a method that trains audio LMs to use their own internal audio representations to perform temporal grounding directly. We introduce a lightweight prediction mechanism trained via two objectives: a binary frame classifier and a novel inhomogeneous Poisson process (IHP) loss that models temporal event intensity. Across word localization, speaker diarization, and event localization tasks, our approach outperforms token-based baselines. Most notably, it achieves a >50x inference speedup and demonstrates robust length generalization, maintaining high accuracy on out-of-distribution audio durations where standard token-based models collapse completely.
Paper Structure (27 sections, 1 theorem, 8 equations, 3 figures, 6 tables)

This paper contains 27 sections, 1 theorem, 8 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Frame-Level Internal Tool Use
  3. Binary Frame-Level Loss
  4. Poisson Process Frame-level Loss
  5. Experimental Setup
  6. Task and Datasets
  7. Modeling
  8. Evaluation Metrics
  9. Results and Discussion
  10. Can Audio LMs Predict Timestamps Zero-Shot?
  11. Fine-Tuning Audio LMs with Frame-Level Tool Use
  12. Generalizing to Multiple Timestamp Generation
  13. Frame-Level Tool Use Enables Efficient Generation
  14. Frame-Level Tool Use Generalizes to Out-of-Distribution Timestamps
  15. Interpolating Token and Poisson Loss
  16. ...and 12 more sections

Key Result

Theorem 2.1

If $t_1, \dots, t_n$ is a realization of an IHP on $[0, T]$ with cumulative hazard $\Lambda(t)$, then the transformed values $z_i = \Lambda(t_i)$ form a unit-rate homogeneous Poisson process on the interval $[0, \Lambda(T)]$.

Figures (3)

  • Figure 1: Frame-level prediction and generative output for predicting timestamps in audio. At inference time, we extract the predicted timestamp from the binary frame-level probabilities $p_1,..,p_T$ via argmax.
  • Figure 2: Inhomogeneous Poisson process frame-level tool training. In the simple case where only 1 timestamp is needed, the $\log\lambda_1,...,\log\lambda_T$ are used to construct a non-parametric probability density estimator over the audio frames, normalized by the total intensity $\Lambda(T) = \sum_{k=1}^T \lambda_k$.
  • Figure 3: Speed-up factor between token-based and Poisson-based timestamp generation across batch sizes. Efficiency gains are larger when the timestamp sequences are long. Poisson-based timestamps are as much $60\times$ faster than token-based timestamps.

Theorems & Definitions (1)

  • Theorem 2.1: Time Rescaling