Audiopedia: Audio QA with Knowledge

TL;DR

Audiopedia tackles the gap in knowledge-intensive audio QA by introducing three sub-tasks—s-AQA, m-AQA, and r-AQA—that require external knowledge linked to named entities in audio. It proposes a two-part framework, AEL and KA$^2$LM, to link entities to a knowledge base and augment large audio-language models with this knowledge, enabling reasoning beyond the audio content. The benchmark uses synthetic data derived from a Wikidata-based KB via TextKVQA, and experiments with three recent LALMs show suboptimal baseline performance that improves substantially with knowledge augmentation. The work provides a publicly available dataset and a generalizable method for integrating knowledge reasoning into audio QA, offering a foundation for more capable audio assistants in real-world, knowledge-rich scenarios. The study also highlights challenges in open-ended answers and retrieval tasks, suggesting directions for expanding domain coverage and multilingual capabilities.

Abstract

In this paper, we introduce Audiopedia, a novel task called Audio Question Answering with Knowledge, which requires both audio comprehension and external knowledge reasoning. Unlike traditional Audio Question Answering (AQA) benchmarks that focus on simple queries answerable from audio alone, Audiopedia targets knowledge-intensive questions. We define three sub-tasks: (i) Single Audio Question Answering (s-AQA), where questions are answered based on a single audio sample, (ii) Multi-Audio Question Answering (m-AQA), which requires reasoning over multiple audio samples, and (iii) Retrieval-Augmented Audio Question Answering (r-AQA), which involves retrieving relevant audio to answer the question. We benchmark large audio language models (LALMs) on these sub-tasks and observe suboptimal performance. To address this, we propose a generic framework that can be adapted to any LALM, equipping them with knowledge reasoning capabilities. Our framework has two components: (i) Audio Entity Linking (AEL) and (ii) Knowledge-Augmented Audio Large Multimodal Model (KA2LM), which together improve performance on knowledge-intensive AQA tasks. To our knowledge, this is the first work to address advanced audio understanding via knowledge-intensive tasks like Audiopedia.

Figures (2)

• Figure 1: Visual Abstract of our proposed task and its sub-tasks: Comparison between traditional AQA (a) and our proposed Audiopedia sub-tasks: s-AQA (b), m-AQA (c), and r-AQA (d). Unlike traditional AQA, our sub-tasks require reasoning over external knowledge linked to the mentioned named entities (shown at the top). Further details are provided in Section \ref{['sec:dataset']}.
• Figure 2: An overview of our proposed methodology: (a) s-AQA: Given an audio, we first perform AEL to obtain the knowledge corresponding to the named entity mentioned in the audio; which is further leveraged by KA$^2$LM to augment a LALM to arrive at an accurate answer. (b) m-AQA: Given multiple audios, we perform AEL on every input audio to obtain their associated knowledge; which is fed to KA$^2$LM along with concatenated input audios, to generate an accurate answer, (c) r-AQA: We solve this task by retrieving relevant audios from the set of the audios first; once retrieved an accurate answer is obtained in a similar fashion as (b). (d) AEL method: We pose the audio entity linking as a ranking problem. Detailed explanation in Section. \ref{['sec:method']}.