Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs

Artem Dementyev, Wazeer Zulfikar, Sinan Hersek, Pascal Getreuer, Anurag Kumar, Vivek Kumar

TL;DR

PhaseCoder tackles the lack of geometry-agnostic spatial audio understanding in multimodal LLMs by introducing a transformer-based encoder that accepts raw multichannel audio plus microphone coordinates to produce spatial tokens. It achieves state-of-the-art microphone-invariant localization on LOCATA, and when integrated with Gemma 3n via a projection layer and LoRA-based fine-tuning, enables the LLM to perform spatial reasoning and targeted transcription. The work demonstrates that spatial embeddings can be effectively fused into LLMs, enabling spatially aware AI across arbitrary arrays and real-world noisy environments. This approach broadens the deployment of spatially aware AI agents for devices with varying microphone geometries.

Abstract

Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over "Spatial Audio Tokens" produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.

PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs

TL;DR

PhaseCoder tackles the lack of geometry-agnostic spatial audio understanding in multimodal LLMs by introducing a transformer-based encoder that accepts raw multichannel audio plus microphone coordinates to produce spatial tokens. It achieves state-of-the-art microphone-invariant localization on LOCATA, and when integrated with Gemma 3n via a projection layer and LoRA-based fine-tuning, enables the LLM to perform spatial reasoning and targeted transcription. The work demonstrates that spatial embeddings can be effectively fused into LLMs, enabling spatially aware AI across arbitrary arrays and real-world noisy environments. This approach broadens the deployment of spatially aware AI agents for devices with varying microphone geometries.

Abstract

Current multimodal LLMs process audio as a mono stream, ignoring the rich spatial information essential for embodied AI. Existing spatial audio models, conversely, are constrained to fixed microphone geometries, preventing deployment across diverse devices. We present PhaseCoder, a transformer-only spatial audio encoder that is agnostic to microphone geometry. PhaseCoder takes raw multichannel audio and microphone coordinates as inputs to perform localization and produces robust spatial embeddings. We demonstrate that Gemma 3n LLM can be fine-tuned to reason over "Spatial Audio Tokens" produced by PhaseCoder. We show our encoder achieves state-of-the-art results on microphone-invariant localization benchmarks and, for the first time, enables an LLM to perform complex spatial reasoning and targeted transcription tasks from an arbitrary microphone array.
Paper Structure (27 sections, 14 equations, 5 figures, 7 tables)

This paper contains 27 sections, 14 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Previous Work
  3. Microphone-Agnostic Localization
  4. Representations of Spatial Audio for LLMs
  5. Methods
  6. PhaseCoder: Spatial Audio Encoder
  7. PhaseCoder Supervised Training
  8. Fine-tuning for Spatial Audio Understanding
  9. Evaluation
  10. PhaseCoder Performance
  11. Metrics.
  12. LLM Spatial Understanding Tasks
  13. Limitations and Future Work
  14. Conclusion
  15. Microphone array coordinate transformations
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Overall system diagram. The spatial audio tokens are injected into the LLM alongside the existing "mono" audio tokens.
  • Figure 2: Visualization of input (raw audio) and classification heads of PhaseCoder. Top: Slice of four microphone channels raw audio, showing phase differences, which can be used to determine the direction of arrival. Bottom: Class probability for distance and azimuth over time.
  • Figure 3: Detailed model architecture. Left: PhaseCoder model architecture and training. The spatial encoder is trained to predict discretized spatial coordinates (azimuth, elevation, distance) using a cross-entropy objective ($\mathcal{L}_{CE}$). For clarity, only two frames from three microphones are shown. Right: Adding spatial tokens to Gemma 3n language model. The PhaseCoder input is added to the existing input pillars (mono audio and text). Down-mixing to mono can be done with various techniques, such as averaging channels or selecting one channel.
  • Figure 4: UMAP projection of PhaseCoder embeddings from the RSL2019 dataset for azimuth angles.
  • Figure 5: UMAP projection of PhaseCoder embeddings from RSL2019 dataset for 150 cm and 100 cm distances.