TIGaussian: Disentangle Gaussians for Spatial-Awared Text-Image-3D Alignment

TL;DR

The paper addresses cross-modal 3D understanding by aligning text, image, and 3D Gaussian Splatting representations. It introduces TIGaussian, featuring a multi-branch 3DGS tokenizer to disentangle Gaussian attributes, a diffusion-enhanced multi-view image fusion to inject 3D context into image features, and a 3D-text projection module for text alignment. Training uses dual contrastive losses to align 3D features with image and text embeddings, achieving state-of-the-art results on Objaverse, ABO, and SUN RGB-D in zero-shot, retrieval, and few-shot tasks. The approach demonstrates that compact, attribute-disentangled 3D tokens can robustly bridge modalities and suggests a scalable path for 3D-aware pretraining.

Abstract

While visual-language models have profoundly linked features between texts and images, the incorporation of 3D modality data, such as point clouds and 3D Gaussians, further enables pretraining for 3D-related tasks, e.g., cross-modal retrieval, zero-shot classification, and scene recognition. As challenges remain in extracting 3D modal features and bridging the gap between different modalities, we propose TIGaussian, a framework that harnesses 3D Gaussian Splatting (3DGS) characteristics to strengthen cross-modality alignment through multi-branch 3DGS tokenizer and modality-specific 3D feature alignment strategies. Specifically, our multi-branch 3DGS tokenizer decouples the intrinsic properties of 3DGS structures into compact latent representations, enabling more generalizable feature extraction. To further bridge the modality gap, we develop a bidirectional cross-modal alignment strategies: a multi-view feature fusion mechanism that leverages diffusion priors to resolve perspective ambiguity in image-3D alignment, while a text-3D projection module adaptively maps 3D features to text embedding space for better text-3D alignment. Extensive experiments on various datasets demonstrate the state-of-the-art performance of TIGaussian in multiple tasks.

Figures (8)

• Figure 1: TIGaussian enables 3D modal pretraining on several tasks, e.g., zero-shot classification, text-3D retrieval and image-3D retrieval. Left: Compared to prior 3D multi-modal alignment methods -- Uni3D and UniGS, TIGaussian presents superior performance on multiple datasets -- Objaverse(-LVIS) and ABO. Right: In challenging scenarios involving ambiguous or complex queries, TIGaussian demonstrates superior performance owing to its disentangled encoder architecture and specialized cross-modal alignment mechanism. We report the similarity score of each item in the figure.
• Figure 2: The TIGaussian framework for text-image-3D tri-modal representation learning. (1) 3DGS Tokenizer: A multi-branch lightweight network decomposes input Gaussians into separate attributes, generating structured latent embedding $F_\mathbb{G}^I$. (2) Image Modality: Multi-view diffusion generates consistent views processed through CLIP to produce 3D-aware visual features $F_\mathbb{I}^{mv}$. (3) Text Modality: The 3D features are projected to text space $F_\mathbb{G}^T$ via a learnable projector. Dual contrastive losses $\mathcal{L}(F_\mathbb{G}^I, F_\mathbb{I}^{mv})$ and $\mathcal{L}(F_\mathbb{G}^T, F_\mathbb{T})$ align all modalities in a shared embedding space, enabling diverse cross-modal applications.
• Figure 3: Illustration of 3DGS tokenizer, where multiple branches are tailored for different attributes for a more compact and effective extraction.
• Figure 4: Illustration of 3D-aware image feature fusion module and 3D-text projector module.
• Figure 5: Few-shot linear probing results on Objaverse-LVIS dataset. 0 number of samples per class stands for zero-shot classification.