SounDiT: Geo-Contextual Soundscape-to-Landscape Generation

TL;DR

This work defines Geo-contextual Soundscape-to-Landscape (GeoS2L) generation, aiming to synthesize geographically coherent landscape images from environmental soundscapes. It introduces SoundingSVI and SonicUrban, two large geo-contextual multimodal datasets, and SounDiT, a diffusion-Transformer model that fuses sound, scene, and landscape cues via a TA-PMoE block and a geo-contextual RAG module. A practical evaluation framework, Place Similarity Score (PSS), measures element-, scene-, and perception-level geographic alignment between input soundscapes and generated landscapes. Empirical results show SounDiT achieves state-of-the-art geo-contextual coherence and visual fidelity across benchmarks, with additional validation from a user study and ablations. The work advances multimodal generation by embedding geographic domain knowledge, enabling applications in geography, urban planning, and environmental science.

Abstract

We present a novel and practically significant problem-Geo-Contextual Soundscape-to-Landscape (GeoS2L) generation-which aims to synthesize geographically realistic landscape images from environmental soundscapes. Prior audio-to-image generation methods typically rely on general-purpose datasets and overlook geographic and environmental contexts, resulting in unrealistic images that are misaligned with real-world environmental settings. To address this limitation, we introduce a novel geo-contextual computational framework that explicitly integrates geographic knowledge into multimodal generative modeling. We construct two large-scale geo-contextual multimodal datasets, SoundingSVI and SonicUrban, pairing diverse soundscapes with real-world landscape images. We propose SounDiT, a novel Diffusion Transformer (DiT)-based model that incorporates geo-contextual scene conditioning to synthesize geographically coherent landscape images. Furthermore, we propose a practically-informed geo-contextual evaluation framework, the Place Similarity Score (PSS), across element-, scene-, and human perception-levels to measure consistency between input soundscapes and generated landscape images. Extensive experiments demonstrate that SounDiT outperforms existing baselines in both visual fidelity and geographic settings. Our work not only establishes foundational benchmarks for GeoS2L generation but also highlights the importance of incorporating geographic domain knowledge in advancing multimodal generative models, opening new directions at the intersection of generative AI, geography, urban planning, and environmental sciences.

Figures (5)

• Figure 1: Geo-contextual soundscape-to-landscape (GeoS2L) generation aims to synthesize realistic landscape images from environmental soundscapes. We introduce large-scale geo-contextual datasets, the SounDiT model, and the Place Similarity Score evaluation framework to support this task.
• Figure 2: Framework of SounDiT. To effectively capture geographic contexts, SounDiT integrates soundscape, scene, and landscape embeddings from pre-trained encoders as inputs. The landscape embedding is fused with the timestep embedding and fed into a self-attention layer to extract visual features. The scene embedding then guides conditioning via a cross-attention layer. The TA-PMoE module further combines these features with soundscape features. During inference, a geo-contextual RAG module provides additional geographic contexts to enhance alignment with real-world environmental settings.
• Figure 3: Representative samples from the SoundingSVI and SonicUrban datasets, showing diverse geographic contexts.
• Figure 4: Geo-contextual evaluation framework. The ground truth and output landscape images are assessed at the element-, scene-, and human-perception levels following geography and urban planning practices.
• Figure 5: Visual comparison of landscape images generated by baseline models (CoDi, Sound2Scene, AudioToken Pre-trained/SD1/SD2, GlueGen) and our proposed SounDiT across SoundingSVI, SonicUrban, and two benchmark datasets. Ground truth landscape images are provided in the last column. Our proposed SounDiT captures significantly more geographic contexts than baselines.