Enhancing Temporal Understanding in Audio Question Answering for Large Audio Language Models

TL;DR

This paper tackles the limited temporal understanding of large audio language models (LALMs) for audio question answering (AQA) and its implications for edge deployment. It introduces a GPT-4–driven TemporalQA data augmentation pipeline to generate reliable temporal questions and answers, and employs curriculum learning to finetune a baseline AQA model with a balanced mix of temporal reasoning and core AQA tasks, guided by metadata and a carefully chosen learning rate. The approach yields significant improvements in temporal reasoning metrics, surpassing baselines on multiple datasets, while also demonstrating practical CPU-based on-device inference. These findings advance commercial viability of AQA on low-power hardware by combining effective data augmentation, targeted fine-tuning, and efficient on-device implementation. The work points to future enhancements in quantization-aware training and evaluation metrics that more accurately capture factual grounding in temporal reasoning.

Abstract

The Audio Question Answering (AQA) task includes audio event classification, audio captioning, and open-ended reasoning. Recently, AQA has garnered attention due to the advent of Large Audio Language Models (LALMs). Current literature focuses on constructing LALMs by integrating audio encoders with text-only Large Language Models (LLMs) through a projection module. While LALMs excel in general audio understanding, they are limited in temporal reasoning, which may hinder their commercial applications and on-device deployment. This paper addresses these challenges and limitations in audio temporal reasoning. First, we introduce a data augmentation technique for generating reliable audio temporal questions and answers using an LLM. Second, we perform a further fine-tuning of an existing baseline using curriculum learning strategy to specialize in temporal reasoning without compromising performance on fine-tuned tasks. We demonstrate the performance of our model using state-of-the-art LALMs on public audio benchmark datasets. Third, we implement our AQA model on-device locally and investigate its CPU inference for edge applications.

Figures (6)

• Figure 1: Our Proposed Framework for Audio Question Answering (AQA) model architecture
• Figure 2: Data Generation for Temporal Reasoning using GPT-4
• Figure 3: Barplot of LTU and GAMA baseline and temporal finetuned checkpoints for temporal dataset.
• Figure 4: I: Graphical User Interface (GUI) of Audio Question Answering on ARM CPU.
• Figure 5: II: Graphical User Interface (GUI) of Audio Question Answering on ARM CPU.