SaLon3R: Structure-aware Long-term Generalizable 3D Reconstruction from Unposed Images

TL;DR

SaLon3R tackles the problem of redundant and inconsistent 3D Gaussian splatting when reconstructing long sequences from unposed images. The method replaces dense pixelwise Gaussians with compact Gaussian anchors via a saliency-aware quantization scheme and learns 3D structure through a lightweight Point Transformer that operates on serialized anchors. Decoding back to Gaussians is adaptive, guided by regionwise saliency, enabling better geometric fidelity and extrapolation in unseen views without test-time camera calibration. Across multiple datasets, SaLon3R achieves online reconstruction at >$10$ FPS, with $50\%$–$90\%$ redundancy removal, and state-of-the-art depth and novel-view synthesis performance, including strong zero-shot generalization.

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have enabled generalizable, on-the-fly reconstruction of sequential input views. However, existing methods often predict per-pixel Gaussians and combine Gaussians from all views as the scene representation, leading to substantial redundancies and geometric inconsistencies in long-duration video sequences. To address this, we propose SaLon3R, a novel framework for Structure-aware, Long-term 3DGS Reconstruction. To our best knowledge, SaLon3R is the first online generalizable GS method capable of reconstructing over 50 views in over 10 FPS, with 50% to 90% redundancy removal. Our method introduces compact anchor primitives to eliminate redundancy through differentiable saliency-aware Gaussian quantization, coupled with a 3D Point Transformer that refines anchor attributes and saliency to resolve cross-frame geometric and photometric inconsistencies. Specifically, we first leverage a 3D reconstruction backbone to predict dense per-pixel Gaussians and a saliency map encoding regional geometric complexity. Redundant Gaussians are compressed into compact anchors by prioritizing high-complexity regions. The 3D Point Transformer then learns spatial structural priors in 3D space from training data to refine anchor attributes and saliency, enabling regionally adaptive Gaussian decoding for geometric fidelity. Without known camera parameters or test-time optimization, our approach effectively resolves artifacts and prunes the redundant 3DGS in a single feed-forward pass. Experiments on multiple datasets demonstrate our state-of-the-art performance on both novel view synthesis and depth estimation, demonstrating superior efficiency, robustness, and generalization ability for long-term generalizable 3D reconstruction. Project Page: https://wrld.github.io/SaLon3R/.

Figures (8)

• Figure 1: Comparison between FreeSplat wang2024freesplat and our SaLon3R. Given a sequence of unposed and uncalibrated images as input, our method achieves online generalizable Gaussian reconstruction with over $10$ FPS speed, allowing high-quality rendering, accurate depth estimation, effectively reducing GS from 9830K to 1043K with nearly 90$\%$ redundancy removal.
• Figure 2: Overview of our proposed SaLon3R. Given every unposed and uncalibrated image as input, an incremental reconstruction network initially predicts the camera parameters $\mathbf{K}_t, \mathbf{P}_t$, pointmaps $\hat{\mathbf{X}}_t$, and pixel-wise Gaussian latents $\hat{\mathbf{G}}_t$. A saliency-aware quantization mechanism is designed to encode dense Gaussians into compact Gaussian anchors, reducing redundancy while maintaining geometric fidelity. A lightweight point transformer is adopted to enable structure-aware Gaussian refinement by capturing the spatial relationships to resolve the inconsistencies and enhance the extrapolation ability, achieving adaptive Gaussian growing with enhanced rendering quality.
• Figure 3: Qualitative Comparison on ScanNet dataset. Given 10 context views as input, we compare the novel view synthesis results for both rendered color and depth. We also provide more qualitative results in the supplementary material.
• Figure 4: Visualization of Global Gaussian Splatting and Extrapolation Views. We zoom in extrapolated views of the bed and desk for better comparison. It demonstrates our method outperforming the FreeSplat with more consistent 3D structure, resulting better rendering performance.
• Figure 5: Effects of Adaptive Gaussian Growing. Applying $\beta$ from $0$ to $0.5$ removes most redundant Gaussians and significantly improves the performance. A larger $\beta=0.8$ could further remove redundant Gaussians with little degradation.