Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mapping

TL;DR

Sat2Sound introduces a unified multimodal framework for zero-shot soundscape mapping that integrates satellite imagery, audio, and text with VLM-generated soundscape captions through a shared codebook of soundscape concepts. By enabling local, patch-level alignment across modalities via a discrete codebook and multi-view metadata, the method achieves state-of-the-art cross-modal retrieval on GeoSound and SoundingEarth and enables a retrieval-based location-specific soundscape synthesis pipeline. Key contributions include (i) a learnable codebook that grounds image patches and audio/text in shared concepts, (ii) a multi-task contrastive objective with pseudo-positives and composed audio queries, and (iii) a scalable soundscape mapping method capable of generating city-, country-, and region-scale maps, plus two training-free synthesis strategies. The approach demonstrates strong retrieval performance, interpretable grounding, and practical synthesis capabilities, with significant implications for AR, urban planning, and geospatial storytelling. The authors provide code and models to facilitate reproducibility and broader adoption.

Abstract

We present Sat2Sound, a multimodal representation learning framework for soundscape mapping, designed to predict the distribution of sounds at any location on Earth. Existing methods for this task rely on satellite image and paired geotagged audio samples, which often fail to capture the diversity of sound sources at a given location. To address this limitation, we enhance existing datasets by leveraging a Vision-Language Model (VLM) to generate semantically rich soundscape descriptions for locations depicted in satellite images. Our approach incorporates contrastive learning across audio, audio captions, satellite images, and satellite image captions. We hypothesize that there is a fixed set of soundscape concepts shared across modalities. To this end, we learn a shared codebook of soundscape concepts and represent each sample as a weighted average of these concepts. Sat2Sound achieves state-of-the-art performance in cross-modal retrieval between satellite image and audio on two datasets: GeoSound and SoundingEarth. Additionally, building on Sat2Sound's ability to retrieve detailed soundscape captions, we introduce a novel application: location-based soundscape synthesis, which enables immersive acoustic experiences. Our code and models will be publicly available.

Figures (5)

• Figure 1: Sat2Sound framework learns a shared multimodal embedding space between satellite images, audio, audio captions, and image captions. Modality-specific encoders generate token embeddings for each modality, which are aligned into a shared codebook through an attention-score-based concept aggregation process.
• Figure 2: (A) Soundscape mapping framework using Sat2Sound's encoders. (B) City-scale soundscape maps using different queries for cities in a) Netherlands; b) USA; c) India. (C) Country-scale soundscape maps created for queries over the USA with a reference land cover map for comparision.
• Figure 3: Alignment between patches in a single image and soundscape concepts in textual query.
• Figure 4: Examples of Top-1 retrieved LLaVA captions for a Bing image by Sat2Sound from our gallery which is the test-set of the GeoSound dataset.
• Figure 5: Some example groups from the GeoSound test set are shown, where each group shares a common set of highly activated codebook concepts, reflecting similar soundscapes of specific geographic areas. The samples in (a) correspond to residential soundscapes, (b) reflect the soundscape of open fields, (c) represent forested area soundscapes, and (d) capture the soundscape of landscapes with nearby water bodies.