HRTFformer: A Spatially-Aware Transformer for Personalized HRTF Upsampling in Immersive Audio Rendering

TL;DR

This work tackles the scalability challenge of personalized HRTFs by proposing HRTFformer, a transformer-based model that operates in the spherical-harmonic domain to upsample sparse HRTF measurements. By mapping measurements to SH coefficients $F^m_l$ and using an encoder–decoder with attention and RoPE positional encoding, the model captures global spatial dependencies while preserving magnitude information; a novel neighbor dissimilarity loss enforces spatial continuity across neighboring directions. The approach achieves state-of-the-art performance on both objective spectral metrics and perceptual localization benchmarks, particularly under extreme sparsity, outperforming algorithmic and learning-based baselines. The method advances immersive audio by enabling scalable, personalized HRTF upsampling, with potential for subjective validation and transfer learning to broaden generalization to unseen listeners.

Abstract

Personalized Head-Related Transfer Functions (HRTFs) are starting to be introduced in many commercial immersive audio applications and are crucial for realistic spatial audio rendering. However, one of the main hesitations regarding their introduction is that creating personalized HRTFs is impractical at scale due to the complexities of the HRTF measurement process. To mitigate this drawback, HRTF spatial upsampling has been proposed with the aim of reducing measurements required. While prior work has seen success with different machine learning (ML) approaches, these models often struggle with long-range spatial consistency and generalization at high upsampling factors. In this paper, we propose a novel transformer-based architecture for HRTF upsampling, leveraging the attention mechanism to better capture spatial correlations across the HRTF sphere. Working in the spherical harmonic (SH) domain, our model learns to reconstruct high-resolution HRTFs from sparse input measurements with significantly improved accuracy. To enhance spatial coherence, we introduce a neighbor dissimilarity loss that promotes magnitude smoothness, yielding more realistic upsampling. We evaluate our method using both perceptual localization models and objective spectral distortion metrics. Experiments show that our model surpasses leading methods by a substantial margin in generating realistic, high-fidelity HRTFs.

Figures (5)

• Figure 1: The HRTF upsampling workflow of HRTFformer: the low-resolution HRTFs are first transformed into the spherical harmonics domain, which provides a compact and physically meaningful representation of directional acoustic information. The resulting SH coefficients serve as the model's input. The encoder extracts global and local spatial features from the input and compresses them into the latent representation, which the decoder then uses to extrapolate and interpolate higher-degree SH coefficients. These are finally converted back into high-resolution HRTFs. This process is applied under four different sparsity conditions to evaluate the model's robustness.
• Figure 2: The model architecture of our HRTFformer. The encoder integrates transformer layers with convolutional downsampling modules to progressively extract and compress spatial features from low-resolution SH coefficients into the latent representation. The decoder combines transformer layers with iterative projection units that perform upsampling.
• Figure 3: LSD distributions for selected subject across various HRTF upsampling methods and sparsity levels. Top to bottom: SYT-FSP-AE, Kalimotxo, IOA3D, and HRTFformer.
• Figure 4: Median plane spectra of example upsampled HRTFs using HRTFformer compared to the original HRTFs for each upsampling factor.
• Figure 5: Upsampled HRTFs for subject P0203 with two sparsity levels, and a source to the right (45° azimuth, 0° elevation). The reference HRTF are shown for comparison.