ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling

TL;DR

The paper addresses the challenge of efficiently constructing an environment acoustic model in unmapped spaces under tight sampling budgets. It introduces ActiveRIR, an audio-visual reinforcement learning policy that navigates and selectively samples observations to build a high-quality acoustic model $f$ (e.g., FS- RIR) with a global information-gain reward, demonstrating substantial reductions in required samples ($N$) and steps ($T$) while maintaining accuracy. Key contributions include a novel Acoustic Prediction reward, integration with a transformer-based acoustic renderer, and empirical evidence of generalization across rendering backends and unseen scenes, outperforming state-of-the-art baselines. The work has practical implications for robotics and augmented/virtual reality where realistic spatial audio with limited data collection is essential, and it opens avenues for acoustic-based scene analysis and 3D reconstruction from acoustic exploration.

Abstract

An environment acoustic model represents how sound is transformed by the physical characteristics of an indoor environment, for any given source/receiver location. Traditional methods for constructing acoustic models involve expensive and time-consuming collection of large quantities of acoustic data at dense spatial locations in the space, or rely on privileged knowledge of scene geometry to intelligently select acoustic data sampling locations. We propose active acoustic sampling, a new task for efficiently building an environment acoustic model of an unmapped environment in which a mobile agent equipped with visual and acoustic sensors jointly constructs the environment acoustic model and the occupancy map on-the-fly. We introduce ActiveRIR, a reinforcement learning (RL) policy that leverages information from audio-visual sensor streams to guide agent navigation and determine optimal acoustic data sampling positions, yielding a high quality acoustic model of the environment from a minimal set of acoustic samples. We train our policy with a novel RL reward based on information gain in the environment acoustic model. Evaluating on diverse unseen indoor environments from a state-of-the-art acoustic simulation platform, ActiveRIR outperforms an array of methods--both traditional navigation agents based on spatial novelty and visual exploration as well as existing state-of-the-art methods.

Figures (5)

• Figure 1: Active acoustic sampling. An agent must intelligently navigate an unmapped 3D scene and actively sample audio-visual observations (the scene's acoustic context) to construct an acoustic model of the environment, given limited navigation time and a fixed sampling budget. When queried with an arbitrary sound source position and receiver pose in the space, the learned environment acoustic model should accurately generate the corresponding Room Impulse Response (RIR) at that pose.
• Figure 2: ActiveRIR policy network architecture and reward. At each step $t$, our policy $\pi$ receives an egocentric visual input $V_t$, the camera pose $P_t$, and the binaural echo response $A_{t-1}$---if it was sampled by the policy at the previous step---and predicts an action $\alpha_t$ that decides both how the agent should move, and if it should sample the current echo response $A_t$. It then uses $A_t$ along with the current visual input $V_t$ to improve its acoustics prediction quality. Given these audio-visual samples ("Context") collected over an episode, the agent uses an off-the-shelf acoustic rendering model to predict the RIR for any arbitrary query pair of sound source and receiver locations. We train our policy with an audio-visual reward which encourages healthy exploration of the scene in search of acoustically important locations, and guides the agent when to sample highly valuable observations, subject to a maximum audio sample budget.
• Figure 3: ActiveRIR vs. Uniform sampling. The ActiveRIR agent (far left) navigates an environment and actively samples observations, collecting context (left) from regions of the environment where acoustics rapidly change---such as in a winding hallway---and which are visually and acoustically distinct from other samples in the context. In contrast, an agent passively sampling at a uniform interval (right) collects an acoustic context (far right) with spatial and visual redundancy, as observed by the bottom two images which show similar views of the same room captured only 1 meter apart.
• Figure 4: Acoustic prediction quality vs timesteps. ActiveRIR (purple) rapidly minimizes STFT error in the acoustic model in fewer steps than an acoustic agent sampling at a fixed interval (orange), and outperforms heuristic approaches as well.
• Figure 5: Active sampling with existing methods.(Left) NAF luo2023learning trained on ActiveRIR-collected context outperforms NAF trained on context collected by a random policy, demonstrating ActiveRIR's ability to select valuable acoustic context agnostic of the acoustic rendering model. (Right) As we grow the context size, ActiveRIR samples high-value observations that rapidly improve global scene acoustic error, producing a final acoustic model with significantly lower error than FS-RIR majumder2022fewshot.