AudioBERT: Audio Knowledge Augmented Language Model
Hyunjong Ok, Suho Yoo, Jaeho Lee
TL;DR
This work investigates whether language models trained on text alone lack auditory commonsense. It introduces AuditoryBench, a benchmark built from a large audio-text corpus to evaluate two tasks: animal sound recognition and sound pitch comparison, revealing that existing LMs perform poorly on auditory knowledge. To address this, the authors propose AudioBERT, a retrieval-augmented framework that detects when auditory knowledge is needed, retrieves relevant audio via CLAP, and injects its embedding into the LM using LoRA adapters, enabling dynamic knowledge augmentation. Experiments show AudioBERT significantly improves performance on AuditoryBench and maintains stability on general tasks, with data quality assessments supporting dataset robustness. The work highlights the potential for cross-modal knowledge augmentation and sets a foundation for auditory-aware language modeling.
Abstract
Recent studies have identified that language models, pretrained on text-only datasets, often lack elementary visual knowledge, \textit{e.g.,} colors of everyday objects. Motivated by this observation, we ask whether a similar shortcoming exists in terms of the \textit{auditory} knowledge. To answer this question, we construct a new dataset called AuditoryBench, which consists of two novel tasks for evaluating auditory knowledge. Based on our analysis using the benchmark, we find that language models also suffer from a severe lack of auditory knowledge. To address this limitation, we propose AudioBERT, a novel method to augment the auditory knowledge of BERT through a retrieval-based approach. First, we detect auditory knowledge spans in prompts to query our retrieval model efficiently. Then, we inject audio knowledge into BERT and switch on low-rank adaptation for effective adaptation when audio knowledge is required. Our experiments demonstrate that AudioBERT is quite effective, achieving superior performance on the AuditoryBench. The dataset and code are available at \bulurl{https://github.com/HJ-Ok/AudioBERT}.