PFGS: Pose-Fused 3D Gaussian Splatting for Complete Multi-Pose Object Reconstruction

TL;DR

PFGS addresses the problem of incomplete 3D object reconstructions when objects are captured in multiple poses. It introduces a pose-aware fusion pipeline that combines a robust global registration stage (including mixed-pose image selection, 3D foundation-model inference, and silhouette-consensus pose fusion) with gradient-based local refinement and 3DGS model completion to integrate auxiliary views into a single, complete 3D Gaussian Splatting representation. The approach yields higher camera pose accuracy and more faithful multi-view reconstructions than strong baselines on both synthetic and real-world data, demonstrating improved occluded-area recovery and novel-view synthesis fidelity. This work enables practical multi-pose scanning workflows for accurate digital twins and robotics by reducing redundant imaging and providing a robust cross-pose alignment strategy.

Abstract

Recent advances in 3D Gaussian Splatting (3DGS) have enabled high-quality, real-time novel-view synthesis from multi-view images. However, most existing methods assume the object is captured in a single, static pose, resulting in incomplete reconstructions that miss occluded or self-occluded regions. We introduce PFGS, a pose-aware 3DGS framework that addresses the practical challenge of reconstructing complete objects from multi-pose image captures. Given images of an object in one main pose and several auxiliary poses, PFGS iteratively fuses each auxiliary set into a unified 3DGS representation of the main pose. Our pose-aware fusion strategy combines global and local registration to merge views effectively and refine the 3DGS model. While recent advances in 3D foundation models have improved registration robustness and efficiency, they remain limited by high memory demands and suboptimal accuracy. PFGS overcomes these challenges by incorporating them more intelligently into the registration process: it leverages background features for per-pose camera pose estimation and employs foundation models for cross-pose registration. This design captures the best of both approaches while resolving background inconsistency issues. Experimental results demonstrate that PFGS consistently outperforms strong baselines in both qualitative and quantitative evaluations, producing more complete reconstructions and higher-fidelity 3DGS models.

Figures (5)

• Figure 1: PFGS: Pose-Fused Gaussian Splatting for Complete Multi-Pose 3D Reconstruction. Starting from a main-pose capture and an auxiliary-pose sequence, our method fuses both into a unified 3D Gaussian Splatting model. PFGS performs robust multi-pose alignment and refinement to integrate diverse viewpoints, completing geometry and appearance across self-occluded and hard-to-see regions. The result is a photorealistic and view-consistent reconstruction that surpasses single-pose limitations.
• Figure 2: System overview of PFGS. Top row: multi-view image sets are captured in a main pose and one or more auxiliary poses; each set is preprocessed with background segmentation and SfM to estimate initial camera poses and sparse geometry. Middle row: we fuse the main pose with the first auxiliary pose via global registration, local registration refinement, and 3DGS completion. Within global registration, we select mixed-pose images from both sets and estimate their cameras using a multi-view 3D foundation model; a silhouette-consensus camera-pose fusion then registers the auxiliary-pose cameras to the main-pose cameras using the mixed-pose cameras as reference. Bottom row: the 3DGS is updated incrementally by repeating this fusion for the first and subsequent auxiliary poses.
• Figure 3: Two-stage silhouette-consensus pose fusion. We align all cameras to the main-pose coordinate frame through two stages. Stage 1: Align the mixed-pose coordinate frame with the main-pose coordinate frame. We select a pair of main-pose cameras from the mixed-pose Fast3R predictions ($P^{main}_{mix}$) and their corresponding cameras from the COLMAP reconstruction ($P_{main}$). A rigid transformation and scale ($d_{main}/d_{mixed}$) are computed from the selected pairs and applied to all Fast3R predictions. The alignment is evaluated by rendering main-pose 3DGS masks using the transformed auxiliary cameras in the mixed-pose group, and computing the average IoU against the corresponding SAM2 masks. The transformation yielding the lowest average IoU is selected. Stage 2: Align the auxiliary-pose coordinate frame with the transformed mixed-pose coordinate frame (now aligned to the main-pose frame) using the same process. This produces a unified multi-pose camera configuration in the main-pose coordinate system for downstream optimization.
• Figure 4: Visual quality of 3DGS reconstructions. Panels show (a) ground-truth images, (b) the main-pose model, and (c) our method. Within each panel, the left image is a rendered view and the right image is a pixel-wise error map, where darker values denote larger $\ell_1$ differences from ground truth. Our reconstruction closely approaches ground truth fidelity while visibly surpassing the main-pose model.
• Figure 5: Visual quality of more 3DGS reconstructions. Panels show (a) ground-truth images, (b) the main-pose model, and (c) our method. Within each panel, the left image is a rendered view and the right image is a pixel-wise error map, where darker values denote larger $\ell_1$ differences from ground truth. Our reconstruction closely approaches ground truth fidelity while visibly surpassing the main-pose model.