Retrieval-Augmented Neural Field for HRTF Upsampling and Personalization

TL;DR

This work tackles the challenge of generating dense HRTF grids from sparse measurements by introducing Retrieval-Augmented Neural Fields (RANF). RANF retrieves $K$ similar subjects and injects their HRTF magnitude $A_{r_k}(\boldsymbol{d})$ and ITD $\tau_{r_k}(\boldsymbol{d})$ as context to a subject-conditioned NF, using a multi-subject Transform-Average-Concatenate (TAC) module to blend information efficiently. The approach extends neural fields with retrieval-based conditioning and LoRA-style parameterizations, achieving superior upsampling performance on the SONICOM dataset and winning Task 2 of the 2024 Listener Acoustic Personalization Challenge. This promises practical gains for immersive binaural rendering by reducing required measurements while preserving localization fidelity, and it provides a scalable framework for subject-specific HRTF personalization.

Abstract

Head-related transfer functions (HRTFs) with dense spatial grids are desired for immersive binaural audio generation, but their recording is time-consuming. Although HRTF spatial upsampling has shown remarkable progress with neural fields, spatial upsampling only from a few measured directions, e.g., 3 or 5 measurements, is still challenging. To tackle this problem, we propose a retrieval-augmented neural field (RANF). RANF retrieves a subject whose HRTFs are close to those of the target subject from a dataset. The HRTF of the retrieved subject at the desired direction is fed into the neural field in addition to the sound source direction itself. Furthermore, we present a neural network that can efficiently handle multiple retrieved subjects, inspired by a multi-channel processing technique called transform-average-concatenate. Our experiments confirm the benefits of RANF on the SONICOM dataset, and it is a key component in the winning solution of Task 2 of the listener acoustic personalization challenge 2024.

Figures (4)

• Figure 1: Overview of RANF. It aims to predict the HRTF magnitude and ITD of a target subject $s$ at a desired direction $\mathbf{d}$ as in 4. We retrieve $K$ subjects in 1 and compute their HRTF magnitude and ITD at the desired direction $\mathbf{d}$ in 2. The retrieved HRTF magnitude and ITD are fed into NF with the sound source direction and the subject-specific parameters selected in 3.
• Figure 2: Network architecture for the proposed RANF. HRTF magnitudes are encoded by a convolution network and decoded by a deconvolution network. The retrieved ITDs are encoded as RFF with the sound source direction and passed to an FC layer. The feature sequence in the gray box is for each of the $K$ retrieved subjects and processed in parallel except for the inter-subject TAC module. The features are aggregated across all retrieved subjects by averaging (Avg.).
• Figure 3: Inter-subject TAC module. Trainable layers are colored, where the layers with the same color share their parameters.
• Figure 4: Ipsilateral HRTF magnitudes for a subject with $(\theta, \phi) = (\pi/2, 0)$.