NAAQA: A Neural Architecture for Acoustic Question Answering

TL;DR

This work tackles Acoustic Question Answering (AQA) by introducing CLEAR2, a more challenging dataset that uses variable-duration scenes and distinct training/test sounds, and proposing NAAQA, an acoustically tailored FiLM-based architecture that leverages 1D time and frequency convolutions. Key innovations include Parallel and Interleaved feature extractors, time and frequency coordinate maps, and targeted complexity optimization that achieves about 79.5% accuracy with roughly 1.68M parameters, significantly fewer than previous VQA models. The study also examines the MALiMo module and demonstrates that while MALiMo offers limited gains on CLEAR2, it yields meaningful improvements on a regenerated DAQA variant (DAQA$^ extprime$), underscoring dataset-dependent benefits. Overall, the paper provides a rigorous analysis of how dataset properties and acoustic priors influence architecture choices for AQA, and it releases code to accelerate research in this nascent area.

Abstract

The goal of the Acoustic Question Answering (AQA) task is to answer a free-form text question about the content of an acoustic scene. It was inspired by the Visual Question Answering (VQA) task. In this paper, based on the previously introduced CLEAR dataset, we propose a new benchmark for AQA, namely CLEAR2, that emphasizes the specific challenges of acoustic inputs. These include handling of variable duration scenes, and scenes built with elementary sounds that differ between training and test set. We also introduce NAAQA, a neural architecture that leverages specific properties of acoustic inputs. The use of 1D convolutions in time and frequency to process 2D spectro-temporal representations of acoustic content shows promising results and enables reductions in model complexity. We show that time coordinate maps augment temporal localization capabilities which enhance performance of the network by ~17 percentage points. On the other hand, frequency coordinate maps have little influence on this task. NAAQA achieves 79.5% of accuracy on the AQA task with ~4 times fewer parameters than the previously explored VQA model. We evaluate the perfomance of NAAQA on an independent data set reconstructed from DAQA. We also test the addition of a MALiMo module in our model on both CLEAR2 and DAQA. We provide a detailed analysis of the results for the different question types. We release the code to produce CLEAR2 as well as NAAQA to foster research in this newly emerging machine learning task.

Figures (4)

• Figure 1: Overview of the CLEAR dataset generation process. Highlighted in red: 10 randomly sampled sounds from the elementary sounds bank, are assembled to create an acoustic scene. The attributes of each elementary sound are depicted in blue. The question template (orange) and the elementary sounds attributes are combined to instantiate a question. The answer is generated by applying each steps of the question functional program (purple) on the acoustic scene definition (blue). The impact of the reverberations can be seen in the changes of the signals envelops.
• Figure 2: Common Architecture. Two inputs: a spectro-temporal representation of an acoustic scene and a textual question. The spectro-temporal representation goes through a feature extractor (Parallel and Interleaved feature extractor detailed in Section \ref{['sec_method_feature_extraction']} for NAAQA and Resnet101 pretrained on ImageNet for Visual FiLM) and then a serie of $J$ Resblocks that are linearly modulated by $\boldsymbol\beta_j$ and $\boldsymbol\gamma_j$ (learned from the question input) via FiLM layers. Coordinate maps are inserted before convolution blocks that are illustrated with a pink border. The output is a probability distribution of the possible answers.
• Figure 3: Acoustic feature extraction
• Figure 4: Test accuracy by question type and the number of relation in the question for Optimized NAAQA Parallel. The overall accuracy for this configuration is 79.1%. The presence of before or after in a question constitutes a temporal relation. The accuracy is N/A for relative position and count compare since these types of question do no include relations. The hyper-parameters are described in the end of Section \ref{['sec:exp_baseline_optimization']}