UniGS: Unified Language-Image-3D Pretraining with Gaussian Splatting

TL;DR

UniGS addresses the limitations of point-cloud based 3D representations in cross-modal pretraining by adopting 3D Gaussian Splatting (3DGS) to model scenes as a set of colored, opacity-bearing Gaussians. It aligns 3DGS with a frozen CLIP space using cross-modal losses $L(T,G)$ and $L(I,G)$ and introduces Gaussian-Aware Guidance to learn fine-grained 3D features via a dual-branch ViT encoder, producing a unified Language-Image-3D latent space. The method ensembles multiple 3DGS datasets and scales the 3D backbone, achieving state-of-the-art results in zero-shot classification, text-driven retrieval, and open-world understanding across several benchmarks, with notable gains such as $L_{CM}(T,I,G)$ improvements and substantial accuracy gains over prior approaches. This work demonstrates the value of 3DGS as a rich, transferable 3D representation for cross-modal learning and provides a scalable path toward broader 3D understanding, albeit with limitations in outdoor scenarios and the need for camera poses in optimization.

Abstract

Recent advancements in multi-modal 3D pre-training methods have shown promising efficacy in learning joint representations of text, images, and point clouds. However, adopting point clouds as 3D representation fails to fully capture the intricacies of the 3D world and exhibits a noticeable gap between the discrete points and the dense 2D pixels of images. To tackle this issue, we propose UniGS, integrating 3D Gaussian Splatting (3DGS) into multi-modal pre-training to enhance the 3D representation. We first rely on the 3DGS representation to model the 3D world as a collection of 3D Gaussians with color and opacity, incorporating all the information of the 3D scene while establishing a strong connection with 2D images. Then, to achieve Language-Image-3D pertaining, UniGS starts with a pre-trained vision-language model to establish a shared visual and textual space through extensive real-world image-text pairs. Subsequently, UniGS employs a 3D encoder to align the optimized 3DGS with the Language-Image representations to learn unified multi-modal representations. To facilitate the extraction of global explicit 3D features by the 3D encoder and achieve better cross-modal alignment, we additionally introduce a novel Gaussian-Aware Guidance module that guides the learning of fine-grained representations of the 3D domain. Through extensive experiments across the Objaverse, ABO, MVImgNet and SUN RGBD datasets with zero-shot classification, text-driven retrieval and open-world understanding tasks, we demonstrate the effectiveness of UniGS in learning a more general and stronger aligned multi-modal representation. Specifically, UniGS achieves leading results across different 3D tasks with remarkable improvements over previous SOTA, Uni3D, including on zero-shot classification (+9.36%), text-driven retrieval (+4.3%) and open-world understanding (+7.92%).

Figures (7)

• Figure 1: The overview of UniGS. UniGS is an innovative, unified, and scalable 3D pretraining framework designed for 3D representation learning. It offers versatile pipelines for various datasets, enabling the efficient 3DGS acquisition to enhance 3D representation learning with SoTA CLIP models. UniGS demonstrates exceptional performance across a broad range of benchmarks.
• Figure 2: Model overview of UniGS. Let $\mu, c, \alpha, s, R$ denote the location, color, opacity, scale, and rotation attribute of 3DGS. (a) Given a 3DGS input, the pre-trained and frozen branch takes 3DGS locations and color as input while the second branch, which is initialized from scratch, focuses on the 3DGS location and the remaining attributes. (b) shows the details of our 3D Encoder and how the prior is leveraged through cross-attention layers.
• Figure 3: Additional ablation study of the quality of 3DGS on the Text-driven retrieval task. The accuracy of Text-driven retrieval on Objaverse under three optimization pipelines.
• Figure 4: Additional ablation study of the quality of 3DGS on the Zero-shot classification task. The accuracy of Zero-shot classification on ABO under three optimization pipelines.
• Figure 5: Left: Uni3D, using 3D point clouds for text-image-3D pre-training. Middle: our UniGS, leveraging 3DGS as the 3D representation with better alignment with image modality. Right: our UniGS learns a more general and stronger multi-modal representation.