Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CrossModalityDiffusion: Multi-Modal Novel View Synthesis with Unified Intermediate Representation

Alex Berian, Daniel Brignac, JhihYang Wu, Natnael Daba, Abhijit Mahalanobis

TL;DR

CrossModalityDiffusion tackles multi-modal novel view synthesis by decoupling encoders, a shared geometry-aware intermediate representation, and modality-specific denoisers within a diffusion framework. It extends GeNVS with modular adapters for EO, SAR, and LiDAR modalities, enabling cross-modality input/output and joint training to enforce consistent geometry across modalities. The approach demonstrates accurate, consistent view synthesis across diverse modalities on ShapeNet Cars data, with benefits from using more views and sensor diversity, while noting substantial computational requirements. This framework enables robust multi-sensor fusion for geospatial imaging and points to future applications beyond MMNVS where modality-agnostic scene understanding is valuable.

Abstract

Geospatial imaging leverages data from diverse sensing modalities-such as EO, SAR, and LiDAR, ranging from ground-level drones to satellite views. These heterogeneous inputs offer significant opportunities for scene understanding but present challenges in interpreting geometry accurately, particularly in the absence of precise ground truth data. To address this, we propose CrossModalityDiffusion, a modular framework designed to generate images across different modalities and viewpoints without prior knowledge of scene geometry. CrossModalityDiffusion employs modality-specific encoders that take multiple input images and produce geometry-aware feature volumes that encode scene structure relative to their input camera positions. The space where the feature volumes are placed acts as a common ground for unifying input modalities. These feature volumes are overlapped and rendered into feature images from novel perspectives using volumetric rendering techniques. The rendered feature images are used as conditioning inputs for a modality-specific diffusion model, enabling the synthesis of novel images for the desired output modality. In this paper, we show that jointly training different modules ensures consistent geometric understanding across all modalities within the framework. We validate CrossModalityDiffusion's capabilities on the synthetic ShapeNet cars dataset, demonstrating its effectiveness in generating accurate and consistent novel views across multiple imaging modalities and perspectives.

CrossModalityDiffusion: Multi-Modal Novel View Synthesis with Unified Intermediate Representation

TL;DR

CrossModalityDiffusion tackles multi-modal novel view synthesis by decoupling encoders, a shared geometry-aware intermediate representation, and modality-specific denoisers within a diffusion framework. It extends GeNVS with modular adapters for EO, SAR, and LiDAR modalities, enabling cross-modality input/output and joint training to enforce consistent geometry across modalities. The approach demonstrates accurate, consistent view synthesis across diverse modalities on ShapeNet Cars data, with benefits from using more views and sensor diversity, while noting substantial computational requirements. This framework enables robust multi-sensor fusion for geospatial imaging and points to future applications beyond MMNVS where modality-agnostic scene understanding is valuable.

Abstract

Geospatial imaging leverages data from diverse sensing modalities-such as EO, SAR, and LiDAR, ranging from ground-level drones to satellite views. These heterogeneous inputs offer significant opportunities for scene understanding but present challenges in interpreting geometry accurately, particularly in the absence of precise ground truth data. To address this, we propose CrossModalityDiffusion, a modular framework designed to generate images across different modalities and viewpoints without prior knowledge of scene geometry. CrossModalityDiffusion employs modality-specific encoders that take multiple input images and produce geometry-aware feature volumes that encode scene structure relative to their input camera positions. The space where the feature volumes are placed acts as a common ground for unifying input modalities. These feature volumes are overlapped and rendered into feature images from novel perspectives using volumetric rendering techniques. The rendered feature images are used as conditioning inputs for a modality-specific diffusion model, enabling the synthesis of novel images for the desired output modality. In this paper, we show that jointly training different modules ensures consistent geometric understanding across all modalities within the framework. We validate CrossModalityDiffusion's capabilities on the synthetic ShapeNet cars dataset, demonstrating its effectiveness in generating accurate and consistent novel views across multiple imaging modalities and perspectives.
Paper Structure (15 sections, 5 equations, 4 figures, 3 tables)

This paper contains 15 sections, 5 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Background: GeNVS
  5. Method
  6. Multi-Modal Dataset
  7. Architecture
  8. Joint Training
  9. Experiments
  10. Single Modality In, Single Modality Out
  11. Range-angle Images
  12. More Images, Better Results
  13. Benefits of Having a Variety of Sensors
  14. Qualitative Discussion
  15. Conclusion

Figures (4)

  • Figure 1: CrossModalityDiffusion input/output examples. Three examples of CrossModalityDiffusion with two input images, and an output from a different modality. From the top down: EO to SAR, LiDAR(RA) to LiDAR(P), LiDAR(P) to EO. The corresponding intermediate feature image is shown in the middle, and the ground truth target image is shown on the right.
  • Figure 2: Dataset Generation. We begin with the SRN-Cars dataset for EO images and corresponding camera pose matrices. We then use the pose matrix and ShapeNet object file for generating LiDAR(RA) and LiDAR(P) images with BLAINDER, and SAR images with RaySAR
  • Figure 3: Architecture of CrossModalityDiffusion.
  • Figure 4: Demonstration of any modality to any modality through unified intermediate representation.