Mix3R: Mixing Feed-forward Reconstruction and Generative 3D Priors for Joint Multi-view Aligned 3D Reconstruction and Pose Estimation

Abstract

Recent trends in sparse-view 3D reconstruction have taken two different paths: feed-forward reconstruction that predicts pixel-aligned point maps without a complete geometry, and generative 3D reconstruction that generates complete geometry but often with poor input-alignment. We present Mix3R, a novel generative 3D reconstruction method which mixes feed-forward reconstruction and 3D generation into a single framework in an aligned manner. Mix3R generates a 3D shape in two stages: a sparse voxel generation stage and a textured geometry generation stage. Unlike pure generative methods, our first-stage generation jointly produces a coarse 3D structure (sparse voxels), per-view point maps and camera parameters aligned to that 3D structure. This is made possible by introducing a Mixture-of-Transformers architecture that inserts global self-attentions to a feed-forward reconstruction model and a 3D generative model, both pretrained on large-scale data. This design effectively retains the pretrained priors but enables better 2D-3D alignment. Based on the initial aligned generations of sparse 3D voxels and point maps, we compute an overlap-based attention bias that is directly added to another pretrained textured geometry generation model, enabling it to correctly place input textures onto generated shapes in a training-free manner. Our design brings mutual benefits to both feed-forward reconstruction and 3D generation: The feed-forward branch learns to ground its predictions to a generative 3D prior, and conversely, the 3D generation branch is conditioned on geometrically informative features from the feed-forward branch. As a result, our method produces 3D shapes with better input alignment compared with pure 3D generative methods, together with camera pose estimations more accurate than previous feed-forward reconstruction methods. Our project page is at https://jsnln.github.io/mix3r/

Figures (10)

• Figure 1: The overall architecture of our two-stage framework. Given multi-view unposed input images, we first employ a mixture-of-transformers architecture that jointly infers a coarse 3D structure, pixel-aligned local point maps, camera poses, and an alignment transformation that aligns point maps to the 3D shape. This alignment is then used to provide fine-grained control in the form of attention bias for the final 3D asset generation.
• Figure 2: The block matching configuration of our MoT architecture. According to different matching types, our network has three different types of mixed blocks.
• Figure 3: Illustrations of different block mixture architectures. Sub-figures (a), (b) and (c) on the left show the structures of the original TRELLIS blocks and $\pi^3$ blocks, whereas (e), (f) and (g) show the three types of mixed blocks obtained from our block matching strategy in Sec. \ref{['sec:method:mot-arch-design']}. Note that we still use residual connections, layer normalization, time step modulation and QK-norm, but do not show them in this figure for simplicity.
• Figure 4: We exhibit the reprojection alignment. Each rendering result is obtained using the decoded 3D Gaussians and the predicted camera parameters.
• Figure 5: Qualitative results of novel-view rendering evaluation. We show input images and novel-view GT images. Our method more accurately restores texture and geometry.