Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation

Chao Yang, Deshui Miao, Chao Tian, Guoqing Zhu, Yameng Gu, Zhenyu He

TL;DR

The paper introduces Infrared-Visible Gaussian Fusion (IVGF), a novel 3D multimodal representation that enables free-viewpoint rendering of fused infrared-visible images. It leverages 3D Gaussian Splatting to model a differentiable radiance field, and introduces a cross-modal adjustment (CMA) module to dynamically modulate Gaussian opacities, resolving cross-modal conflicts during rendering. A two-stage training regime combines modality-specific reconstruction with a fusion loss that jointly optimizes intensity and gradient fidelity, preserving salient features from both infrared and visible modalities. Experimental results on the RGBT-Scenes dataset show that IVGF outperforms 2D render-then-fusion baselines in SSIM and LPIPS, while offering efficient rendering and improved informational completeness in fused scenes.

Abstract

Infrared-visible image fusion aims to integrate infrared and visible information into a single fused image. Existing 2D fusion methods focus on fusing images from fixed camera viewpoints, neglecting a comprehensive understanding of complex scenarios, which results in the loss of critical information about the scene. To address this limitation, we propose a novel Infrared-Visible Gaussian Fusion (IVGF) framework, which reconstructs scene geometry from multimodal 2D inputs and enables direct rendering of fused images. Specifically, we propose a cross-modal adjustment (CMA) module that modulates the opacity of Gaussians to solve the problem of cross-modal conflicts. Moreover, to preserve the distinctive features from both modalities, we introduce a fusion loss that guides the optimization of CMA, thus ensuring that the fused image retains the critical characteristics of each modality. Comprehensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.

Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation

TL;DR

The paper introduces Infrared-Visible Gaussian Fusion (IVGF), a novel 3D multimodal representation that enables free-viewpoint rendering of fused infrared-visible images. It leverages 3D Gaussian Splatting to model a differentiable radiance field, and introduces a cross-modal adjustment (CMA) module to dynamically modulate Gaussian opacities, resolving cross-modal conflicts during rendering. A two-stage training regime combines modality-specific reconstruction with a fusion loss that jointly optimizes intensity and gradient fidelity, preserving salient features from both infrared and visible modalities. Experimental results on the RGBT-Scenes dataset show that IVGF outperforms 2D render-then-fusion baselines in SSIM and LPIPS, while offering efficient rendering and improved informational completeness in fused scenes.

Abstract

Infrared-visible image fusion aims to integrate infrared and visible information into a single fused image. Existing 2D fusion methods focus on fusing images from fixed camera viewpoints, neglecting a comprehensive understanding of complex scenarios, which results in the loss of critical information about the scene. To address this limitation, we propose a novel Infrared-Visible Gaussian Fusion (IVGF) framework, which reconstructs scene geometry from multimodal 2D inputs and enables direct rendering of fused images. Specifically, we propose a cross-modal adjustment (CMA) module that modulates the opacity of Gaussians to solve the problem of cross-modal conflicts. Moreover, to preserve the distinctive features from both modalities, we introduce a fusion loss that guides the optimization of CMA, thus ensuring that the fused image retains the critical characteristics of each modality. Comprehensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.
Paper Structure (11 sections, 9 equations, 4 figures, 2 tables)

This paper contains 11 sections, 9 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminaries: 3D Gaussian Splatting
  4. Infrared-Visible Gaussian Representation
  5. Loss Function
  6. Experiments
  7. Experimental Setup
  8. Qualitative Analysis of Fusion Methods
  9. Quantitative Analysis of Fusion Methods
  10. Ablation Study
  11. Conclusion

Figures (4)

  • Figure 1: 3D Gaussian scene representation. (a) Visible 3D scene. (b) Infrared 3D scene. (c) Multimodal 3D scene. (d) Our Infrared-Visible Gaussian Fusion (IVGF) scene. Direct rendering from multimodal scenes introduces cumulative errors, leading to the loss of detail and infrared information.
  • Figure 2: Overview of our Infrared-Visible Gaussian Fusion (IVGF) framework. First, infrared and visible Gaussians are trained separately. Subsequently, they are concatenated and processed by a cross-modal adjustment (CMA) module, which refines the parameters of Gaussians to reconstruct a fused 3D scene.
  • Figure 3: Qualitative results of different methods. The first box contains infrared and visible images, the second box shows the results of multimodal rendering, the third box presents the fusion results from the render-then-fusion methods, and the final box displays our results.
  • Figure 4: Ablation study on the effects of different modules. The cross-modal adjustment (CMA) module resolves cross-modal error accumulations, while the proposed fusion loss ensures the retention of essential features from both modalities in the rendered results.