Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uni3R: Unified 3D Reconstruction and Semantic Understanding via Generalizable Gaussian Splatting from Unposed Multi-View Images

Xiangyu Sun, Haoyi Jiang, Liu Liu, Seungtae Nam, Gyeongjin Kang, Xinjie Wang, Wei Sui, Zhizhong Su, Wenyu Liu, Xinggang Wang, Eunbyung Park

Abstract

Reconstructing and semantically interpreting 3D scenes from sparse 2D views remains a fundamental challenge in computer vision. Conventional methods often decouple semantic understanding from reconstruction or necessitate costly per-scene optimization, thereby restricting their scalability and generalizability. In this paper, we introduce Uni3R, a novel feed-forward framework that jointly reconstructs a unified 3D scene representation enriched with open-vocabulary semantics, directly from unposed multi-view images. Our approach leverages a Cross-View Transformer to robustly integrate information across arbitrary multi-view inputs, which then regresses a set of 3D Gaussian primitives endowed with semantic feature fields. This unified representation facilitates high-fidelity novel view synthesis, open-vocabulary 3D semantic segmentation, and depth prediction, all within a single, feed-forward pass. Extensive experiments demonstrate that Uni3R establishes a new state-of-the-art across multiple benchmarks, including 25.07 PSNR on RE10K and 55.84 mIoU on ScanNet. Our work signifies a novel paradigm towards generalizable, unified 3D scene reconstruction and understanding. The code is available at https://github.com/HorizonRobotics/Uni3R.

Uni3R: Unified 3D Reconstruction and Semantic Understanding via Generalizable Gaussian Splatting from Unposed Multi-View Images

Abstract

Reconstructing and semantically interpreting 3D scenes from sparse 2D views remains a fundamental challenge in computer vision. Conventional methods often decouple semantic understanding from reconstruction or necessitate costly per-scene optimization, thereby restricting their scalability and generalizability. In this paper, we introduce Uni3R, a novel feed-forward framework that jointly reconstructs a unified 3D scene representation enriched with open-vocabulary semantics, directly from unposed multi-view images. Our approach leverages a Cross-View Transformer to robustly integrate information across arbitrary multi-view inputs, which then regresses a set of 3D Gaussian primitives endowed with semantic feature fields. This unified representation facilitates high-fidelity novel view synthesis, open-vocabulary 3D semantic segmentation, and depth prediction, all within a single, feed-forward pass. Extensive experiments demonstrate that Uni3R establishes a new state-of-the-art across multiple benchmarks, including 25.07 PSNR on RE10K and 55.84 mIoU on ScanNet. Our work signifies a novel paradigm towards generalizable, unified 3D scene reconstruction and understanding. The code is available at https://github.com/HorizonRobotics/Uni3R.

Paper Structure

This paper contains 32 sections, 10 equations, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Differentiable Neural Representations
  4. Feed-forward 3D Reconstruction
  5. Open-Vocabulary Segmentation in 3DGS
  6. Method
  7. Feed-Forward Gaussian Splatting
  8. Intrinsic Embedding
  9. Cross-View Transformer Encoder
  10. Decoding Gaussian Parameters
  11. Rendering with Open-Vocabulary Semantics
  12. Training Objectives
  13. Photometric Loss ($\mathcal{L}_{\text{rgb}}$).
  14. Semantic Loss ($\mathcal{L}_{\text{sem}}$).
  15. Geometry Loss ($\mathcal{L}_{\text{geo}}$).
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Uni3R takes unposed arbitrary multi-view images as input and produces a unified 3D Gaussian scene representation, enabling state-of-the-art performance in view synthesis, semantic segmentation, and depth estimation within a single forward pass.
  • Figure 2: Architectural overview of the Uni3R pipeline. Uni3R predicts a set of Gaussian primitives with jointly integrated geometry, appearance, and open-vocabulary semantics in a single pass, eliminating the need for per-scene optimization.
  • Figure 3: Qualitative comparison of novel view synthesis on RealEstate10k test set with 8 input images.
  • Figure 4: Qualitative Comparison of Novel-View Segmentation on ScanNet.
  • Figure 5: Model training w/ and w/o geo. loss on 4 views.