Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Jihoon Jeong, Pooneh Mousavi, Mirco Ravanelli, Cem Subakan

Abstract

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chains remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.

Listen First, Then Answer: Timestamp-Grounded Speech Reasoning

Abstract

Large audio-language models (LALMs) can generate reasoning chains for their predictions, but it remains unclear whether these reasoning chains remain grounded in the input audio. In this paper, we propose an RL-based strategy that grounds the reasoning outputs of LALMs with explicit timestamp annotations referring to relevant segments of the audio signal. Our analysis shows that timestamp grounding leads the model to attend more strongly to audio tokens during reasoning generation. Experiments on four speech-based benchmark datasets demonstrate that our approach improves performance compared to both zero-shot reasoning and fine-tuning without timestamp grounding. Additionally, grounding amplifies desirable reasoning behaviors, such as region exploration, audiology verification, and consistency, underscoring the importance of grounding mechanisms for faithful multimodal reasoning.
Paper Structure (17 sections, 3 equations, 5 figures, 2 tables)

This paper contains 17 sections, 3 equations, 5 figures, 2 tables.

Table of Contents

  1. Attention Map Analysis Details
  2. Layer-wise Aggregated Attention Analysis Across Query Types
  3. Case Study: Attention Aggregation during Timestamp Generation
  4. Layer-wise Attention Statistics for Timestamp Generation
  5. Additional Experimental Settings
  6. Prompt Details
  7. Stage 1: Supervised Timestamp Alignment for Qwen2.5-Omni
  8. Stage 1: Supervised Timestamp Alignment for Audio Flamingo 3
  9. Stage 2: Timestamp-Grounded Reasoning Instruction
  10. Training Implementation Details
  11. Additional Methodological Details
  12. Problem Setup
  13. Group Relative Policy Optimization (GRPO)
  14. RL Reward Motivation
  15. Behavior Analysis Metrics
  16. ...and 2 more sections

Figures (5)

  • Figure 5: Layer-wise attention aggregation across different query types. Unlike Fig. \ref{['fig:audio_attention_blocks']}, the values shown here correspond to summed attention scores within each semantic block rather than per-token normalized values. The overall attention profiles remain highly consistent across layers, indicating that LALMs follow a similar internal processing pattern for audio-conditioned reasoning tasks.
  • Figure 6: Block-level attention aggregation at Layer 14 during timestamp generation. Attention weights from each output token are aggregated across four semantic input blocks. Tokens corresponding to timestamp prediction (highlighted in a red box) allocate increased attention to the audio block, indicating that temporal reasoning relies on acoustic evidence.
  • Figure 7: Layer-wise attention aggregation during timestamp generation across two LALM architectures. (a) Audio Flamingo 3 and (b) Qwen2.5-Omni. Despite architectural differences, both models exhibit a similar shift toward increased reliance on acoustic representations during timestamp generation, supporting the tendency discussed in the main text.
  • Figure 8: Comparison of reasoning traces between vanilla GRPO and our method on the MMAU dataset - 1.
  • Figure 9: Comparison of reasoning traces between vanilla GRPO and our method on the MMAU dataset - 2.