GeoLRM: Geometry-Aware Large Reconstruction Model for High-Quality 3D Gaussian Generation

TL;DR

GeoLRM tackles high-quality 3D Gaussian generation from multi-view images by introducing a geometry-aware large reconstruction model that uses a 3D-aware transformer with deformable cross-attention. It employs a two-stage pipeline—first predicting a sparse $16^3$ occupancy grid of 3D anchors, then refining them into dense Gaussians (up to $512k$) using projection-aware fusion—to achieve detailed 3D content from up to $21$ views within $11$ GB of memory. The model combines a hierarchical image encoder with 3D Rotary Positional Embeddings and deformable cross-attention to efficiently fuse 2D features into 3D space and produce Gaussians via an MLP decoder. Trained on $GObjaverse$ and evaluated on $GSO$ and OmniObject3D, GeoLRM attains state-of-the-art results across multiple metrics, especially with dense inputs, and demonstrates strong scalability for practical 3D content creation and potential video applications.

Abstract

In this work, we introduce the Geometry-Aware Large Reconstruction Model (GeoLRM), an approach which can predict high-quality assets with 512k Gaussians and 21 input images in only 11 GB GPU memory. Previous works neglect the inherent sparsity of 3D structure and do not utilize explicit geometric relationships between 3D and 2D images. This limits these methods to a low-resolution representation and makes it difficult to scale up to the dense views for better quality. GeoLRM tackles these issues by incorporating a novel 3D-aware transformer structure that directly processes 3D points and uses deformable cross-attention mechanisms to effectively integrate image features into 3D representations. We implement this solution through a two-stage pipeline: initially, a lightweight proposal network generates a sparse set of 3D anchor points from the posed image inputs; subsequently, a specialized reconstruction transformer refines the geometry and retrieves textural details. Extensive experimental results demonstrate that GeoLRM significantly outperforms existing models, especially for dense view inputs. We also demonstrate the practical applicability of our model with 3D generation tasks, showcasing its versatility and potential for broader adoption in real-world applications. The project page: https://linshan-bin.github.io/GeoLRM/.

Figures (7)

• Figure 1: Image to 3D using GeoLRM. Initially, a 3D-aware diffusion model, specifically SV3D sv3d, transforms an input image into multiple views. Subsequently, these views are processed by our GeoLRM to generate detailed 3D assets. Unlike other LRM-based approaches, GeoLRM notably improves as the number of input views increases.
• Figure 2: Pipeline of the proposed GeoLRM, a geometry-powered method for efficient image to 3D reconstruction. The process begins with the transformation of dense tokens into an occupancy grid via a Proposal Transformer, which captures spatial occupancy from hierarchical image features extracted using a combination of a convolutional layer and DINOv2 dinov2. Sparse tokens representing occupied voxels are further processed through a Reconstruction Transformer that employs self-attention and deformable cross-attention mechanisms to refine geometry and retrieve texture details with 3D to 2D projection. Finally, the refined 3D tokens are converted into 3D Gaussians for real-time rendering.
• Figure 3: Qualitative comparisons of different image-3D methods. Better viewed when zoomed in.
• Figure 4: Qualitative comparison concerning scalability in input views.
• Figure 5: Effects of excluding high-level and low-level features in the image encoder.