Spectral Probing of Feature Upsamplers in 2D-to-3D Scene Reconstruction

TL;DR

The results indicate that reconstruction quality is more closely related to preserving spectral structure than to enhancing spatial detail, highlighting spectral consistency as an important principle for designing upsampling strategies in 2D-to-3D pipelines.

Abstract

A typical 2D-to-3D pipeline takes multi-view images as input, where a Vision Foundation Model (VFM) extracts features that are spatially upsampled to dense representations for 3D reconstruction. If dense features across views preserve geometric consistency, differentiable rendering can recover an accurate 3D representation, making the feature upsampler a critical component. Recent learnable upsampling methods mainly aim to enhance spatial details, such as sharper geometry or richer textures, yet their impact on 3D awareness remains underexplored. To address this gap, we introduce a spectral diagnostic framework with six complementary metrics that characterize amplitude redistribution, structural spectral alignment, and directional stability. Across classical interpolation and learnable upsampling methods on CLIP and DINO backbones, we observe three key findings. First, structural spectral consistency (SSC/CSC) is the strongest predictor of NVS quality, whereas High-Frequency Spectral Slope Drift (HFSS) often correlates negatively with reconstruction performance, indicating that emphasizing high-frequency details alone does not necessarily improve 3D reconstruction. Second, geometry and texture respond to different spectral properties: Angular Energy Consistency (ADC) correlates more strongly with geometry-related metrics, while SSC/CSC influence texture fidelity slightly more than geometric accuracy. Third, although learnable upsamplers often produce sharper spatial features, they rarely outperform classical interpolation in reconstruction quality, and their effectiveness depends on the reconstruction model. Overall, our results indicate that reconstruction quality is more closely related to preserving spectral structure than to enhancing spatial detail, highlighting spectral consistency as an important principle for designing upsampling strategies in 2D-to-3D pipelines.

Figures (14)

• Figure 1: Overview of the spectral probing pipeline. Feature upsamplers reshape spectral structure, which in turn affects reconstruction quality. Our framework measures this relationship through six spectral diagnostics and NVS evaluation. Specifically, multi-view images are resized to $H\times W$ and encoded into patch-grid features ($h\times w$). Different upsampling strategies produce dense maps at $H\times W$, which are used to regress 3D Gaussian parameters via differentiable rendering. Spectral changes are correlated with NVS quality (PSNR, SSIM, LPIPS) to probe 3D awareness.
• Figure 2: NVS visualizations using CLIP+DUSt3R wang2024dust3r. Classical interpolation methods often achieve performance comparable to learned upsamplers. Best and second-best results are shown in bold and underlined, respectively.
• Figure 3: Spectral–reconstruction correlations. Each row shows the Spearman correlation heatmap. Results of other upsamplers are provided in the Supplementary Material.
• Figure 4: Spectral diagnostics under geometry-only (AG) and texture-only (AT) settings (CLIP+DUSt3R wang2024dust3r). Additional results are in the Supplementary Material.
• Figure 6: Spectral–reconstruction correlations. Each row shows the Spearman correlation heatmap.