InsideOut: Integrated RGB-Radiative Gaussian Splatting for Comprehensive 3D Object Representation

TL;DR

InsideOut addresses the challenge of fusing RGB surface detail with subsurface X-ray structure in 3D representations. It extends $3DGS$ by introducing radiative splatting for X-ray data, a hierarchical fitting pipeline for cross-modal geometric alignment, and a novel X-ray reference loss guided by cross-sectional slices. A paired RGB–X-ray dataset is collected to train and evaluate the method, showing improved internal detail, sharper layer boundaries, and coherent surface rendering. The approach broadens the applicability of differentiable 3D splatting to domains requiring both external appearance and internal structure, such as medical diagnostics, cultural heritage, and manufacturing.

Abstract

We introduce InsideOut, an extension of 3D Gaussian splatting (3DGS) that bridges the gap between high-fidelity RGB surface details and subsurface X-ray structures. The fusion of RGB and X-ray imaging is invaluable in fields such as medical diagnostics, cultural heritage restoration, and manufacturing. We collect new paired RGB and X-ray data, perform hierarchical fitting to align RGB and X-ray radiative Gaussian splats, and propose an X-ray reference loss to ensure consistent internal structures. InsideOut effectively addresses the challenges posed by disparate data representations between the two modalities and limited paired datasets. This approach significantly extends the applicability of 3DGS, enhancing visualization, simulation, and non-destructive testing capabilities across various domains.

Figures (7)

• Figure 1: Applications of InsideOut. Our approach visualizes external appearance and internal structures, enabling analysis and physical simulation.
• Figure 2: Framework of InsideOut. Our framework employs a five-stage pipeline to integrate RGB and X-ray images into a unified 3D representation. Starting with point cloud initialization from both modalities, the pipeline performs surface rendering to learn basic structures, hierarchical fitting to achieve geometric alignment, internal rendering with cross-sectional guidance for structural refinement, and visual alignment to transfer colors and details, ultimately producing a comprehensive 3D model capturing both external appearance and internal structures.
• Figure 3: Overview of hierarchical fitting. The hierarchical fitting aligns RGB and radiative Gaussian point clouds through three progressive stages: global fitting for overall alignment, local fitting with clustering at multiple resolutions, and fine fitting for individual Gaussian alignment.
• Figure 4: Qualitative comparison of cross-sectional 3D models. Coronal slices (upper) and axial slices (lower) are listed.
• Figure 5: Comparison of surface and internal structure in mesh reconstruction. Mesh reconstruction quality is compared across multiple views: front view, surface zoom, side view, and X-ray shader rendering. Pink boxes indicate zoomed regions. Normal orientation histograms below each method illustrate the distribution of surface normals.