Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Omnidirectional: Neural Ambisonics Encoding for Arbitrary Microphone Directivity Patterns using Cross-Attention

Mikko Heikkinen, Archontis Politis, Konstantinos Drossos, Tuomas Virtanen

TL;DR

This work tackles the problem of encoding microphone array signals into Ambisonics for arbitrary, non-ideal arrays by leveraging directional array transfer functions (ATFs) rather than relying solely on geometry. It introduces a neural architecture with separate encoders for audio signals and ATFs, fused through cross-attention to produce an array-independent latent representation that decodes into a mixing matrix $oldsymbol{E}$, enabling $oldsymbol{b}_{t,f}\u223c oldsymbol{E}oldsymbol{x}_{t,f}$. The method is evaluated on simulated mobile-phone scattering and free-field conditions, showing superior SI-SDR and competitive binaural and spectral metrics compared with static encoders, parametric DSP, and a prior DNN model, while using a relatively compact parameter count. This cross-attention approach offers a practical path to robust Ambisonics encoding for consumer devices with complex directivity and scattering, with potential for perceptual validation and larger-capacity extensions in future work.

Abstract

We present a deep neural network approach for encoding microphone array signals into Ambisonics that generalizes to arbitrary microphone array configurations with fixed microphone count but varying locations and frequency-dependent directional characteristics. Unlike previous methods that rely only on array geometry as metadata, our approach uses directional array transfer functions, enabling accurate characterization of real-world arrays. The proposed architecture employs separate encoders for audio and directional responses, combining them through cross-attention mechanisms to generate array-independent spatial audio representations. We evaluate the method on simulated data in two settings: a mobile phone with complex body scattering, and a free-field condition, both with varying numbers of sound sources in reverberant environments. Evaluations demonstrate that our approach outperforms both conventional digital signal processing-based methods and existing deep neural network solutions. Furthermore, using array transfer functions instead of geometry as metadata input improves accuracy on realistic arrays.

Beyond Omnidirectional: Neural Ambisonics Encoding for Arbitrary Microphone Directivity Patterns using Cross-Attention

TL;DR

This work tackles the problem of encoding microphone array signals into Ambisonics for arbitrary, non-ideal arrays by leveraging directional array transfer functions (ATFs) rather than relying solely on geometry. It introduces a neural architecture with separate encoders for audio signals and ATFs, fused through cross-attention to produce an array-independent latent representation that decodes into a mixing matrix , enabling . The method is evaluated on simulated mobile-phone scattering and free-field conditions, showing superior SI-SDR and competitive binaural and spectral metrics compared with static encoders, parametric DSP, and a prior DNN model, while using a relatively compact parameter count. This cross-attention approach offers a practical path to robust Ambisonics encoding for consumer devices with complex directivity and scattering, with potential for perceptual validation and larger-capacity extensions in future work.

Abstract

We present a deep neural network approach for encoding microphone array signals into Ambisonics that generalizes to arbitrary microphone array configurations with fixed microphone count but varying locations and frequency-dependent directional characteristics. Unlike previous methods that rely only on array geometry as metadata, our approach uses directional array transfer functions, enabling accurate characterization of real-world arrays. The proposed architecture employs separate encoders for audio and directional responses, combining them through cross-attention mechanisms to generate array-independent spatial audio representations. We evaluate the method on simulated data in two settings: a mobile phone with complex body scattering, and a free-field condition, both with varying numbers of sound sources in reverberant environments. Evaluations demonstrate that our approach outperforms both conventional digital signal processing-based methods and existing deep neural network solutions. Furthermore, using array transfer functions instead of geometry as metadata input improves accuracy on realistic arrays.
Paper Structure (11 sections, 5 equations, 3 figures, 1 table)

This paper contains 11 sections, 5 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Signal model and encoding task
  4. Method
  5. Evaluation
  6. Baseline Methods
  7. Data
  8. Hyperparameters
  9. Metrics
  10. Results and Discussion
  11. Conclusion

Figures (3)

  • Figure 1: Sketch of the proposed method. The inputs $\mathbf{X}$ and $\mathbf{H}$ are encoded into latent representations $\mathbf{Z_X}$ and $\mathbf{Z_H}$ respectively, which are then combined through an attention mechanism to produce $\mathbf{Z_{Attn}}$. Finally, $\mathbf{Z_{Attn}}$ is decoded to generate the mixing matrix $\mathbf{E}$
  • Figure 2: Attention weights averaged over time and frequency with a sound source at 0° azimuth and 90° colatitude in an anechoic room.
  • Figure 3: Magnitude response errors on the mobile dataset averaged over Ambisonics channels.