Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model

Mojtaba Heydari, Mehrez Souden, Bruno Conejo, Joshua Atkins

TL;DR

ImmerseDiffusion tackles generating 3D immersive spatial audio by mapping FOA to a latent space via an ambisonic autoencoder and diffusing through a Transformer-based latent diffusion model. It introduces two conditioning streams—descriptive prompts with spatial/environmental text via ELSA and parametric prompts with explicit spatial/environmental parameters via CLAP—along with temporal conditioning, enabling both narrative and parameter-driven generation. The paper contributes a FOA-focused spatial codec with a continuous VAE bottleneck, a DiT-based diffusion model, and novel evaluation metrics (Ambisonics FAD, spatial KL, and CLAP-based scores) to quantify audio quality and spatial fidelity. Experimental results on spatialized datasets show competitive FOA reconstruction and robust spatial localization under both conditioning modes, highlighting the method's potential for VR/AR and interactive audio workflows. The work's significance lies in enabling end-to-end, controllable generation of 3D soundscapes conditioned on space and time, with practical implications for immersive media, gaming, and simulation environments.

Abstract

We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects. ImmerseDiffusion is trained to generate first-order ambisonics (FOA) audio, which is a conventional spatial audio format comprising four channels that can be rendered to multichannel spatial output. The proposed generative system is composed of a spatial audio codec that maps FOA audio to latent components, a latent diffusion model trained based on various user input types, namely, text prompts, spatial, temporal and environmental acoustic parameters, and optionally a spatial audio and text encoder trained in a Contrastive Language and Audio Pretraining (CLAP) style. We propose metrics to evaluate the quality and spatial adherence of the generated spatial audio. Finally, we assess the model performance in terms of generation quality and spatial conformance, comparing the two proposed modes: ``descriptive", which uses spatial text prompts) and ``parametric", which uses non-spatial text prompts and spatial parameters. Our evaluations demonstrate promising results that are consistent with the user conditions and reflect reliable spatial fidelity.

ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model

TL;DR

ImmerseDiffusion tackles generating 3D immersive spatial audio by mapping FOA to a latent space via an ambisonic autoencoder and diffusing through a Transformer-based latent diffusion model. It introduces two conditioning streams—descriptive prompts with spatial/environmental text via ELSA and parametric prompts with explicit spatial/environmental parameters via CLAP—along with temporal conditioning, enabling both narrative and parameter-driven generation. The paper contributes a FOA-focused spatial codec with a continuous VAE bottleneck, a DiT-based diffusion model, and novel evaluation metrics (Ambisonics FAD, spatial KL, and CLAP-based scores) to quantify audio quality and spatial fidelity. Experimental results on spatialized datasets show competitive FOA reconstruction and robust spatial localization under both conditioning modes, highlighting the method's potential for VR/AR and interactive audio workflows. The work's significance lies in enabling end-to-end, controllable generation of 3D soundscapes conditioned on space and time, with practical implications for immersive media, gaming, and simulation environments.

Abstract

We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects. ImmerseDiffusion is trained to generate first-order ambisonics (FOA) audio, which is a conventional spatial audio format comprising four channels that can be rendered to multichannel spatial output. The proposed generative system is composed of a spatial audio codec that maps FOA audio to latent components, a latent diffusion model trained based on various user input types, namely, text prompts, spatial, temporal and environmental acoustic parameters, and optionally a spatial audio and text encoder trained in a Contrastive Language and Audio Pretraining (CLAP) style. We propose metrics to evaluate the quality and spatial adherence of the generated spatial audio. Finally, we assess the model performance in terms of generation quality and spatial conformance, comparing the two proposed modes: ``descriptive", which uses spatial text prompts) and ``parametric", which uses non-spatial text prompts and spatial parameters. Our evaluations demonstrate promising results that are consistent with the user conditions and reflect reliable spatial fidelity.

Paper Structure

This paper contains 11 sections, 8 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Spatial Audio Format
  4. Spatial Codec
  5. Spatial, Temporal and Environmental Conditioning
  6. Diffusion Model
  7. Datasets
  8. Training
  9. Evaluation Metrics
  10. Results and Discussions
  11. Conclusion

Figures (1)

  • Figure 1: (I) Descriptive Conditioner: integrates ELSA devnani2024elsa text conditioner to encode prompts detailing the audio source and spatial and environmental context plus temporal conditioning blocks. (II) Parametric Conditioner: including LAION wu2023large text encoder to provide non-spatial text embeddings along with parametric spatial, environmental, and temporal conditioning blocks. (III) ImmerseDiffusion overall architecture, comprising the ambisonic autoencoder, conditioner, and transformer diffusion including training and inference diagrams.