FUSER: Feed-Forward MUltiview 3D Registration Transformer and SE(3)$^N$ Diffusion Refinement

TL;DR

FUSER introduces a novel feed-forward multiview registration transformer that jointly encodes all scans to predict global poses, addressing global-context limitations and inefficiencies of pairwise pipelines. It combines Absolute Geometric Encoding, Geometric Alternating Attention with transferred 2D attention priors, and a semi-supervised relative-pose supervision strategy, achieving high accuracy and efficiency. Building on FUSER, FUSER-DF formulates multiview pose refinement as a prior-aware SE(3)$^N$ diffusion process, using FUSER as a surrogate denoiser and deriving a variational lower bound to supervise denoising. Extensive experiments on 3DMatch, ScanNet, and ArkitScenes demonstrate state-of-the-art performance with orders-of-magnitude speedups, validating the approach's practicality and scalability for real-world 3D reconstruction tasks.

Abstract

Registration of multiview point clouds conventionally relies on extensive pairwise matching to build a pose graph for global synchronization, which is computationally expensive and inherently ill-posed without holistic geometric constraints. This paper proposes FUSER, the first feed-forward multiview registration transformer that jointly processes all scans in a unified, compact latent space to directly predict global poses without any pairwise estimation. To maintain tractability, FUSER encodes each scan into low-resolution superpoint features via a sparse 3D CNN that preserves absolute translation cues, and performs efficient intra- and inter-scan reasoning through a Geometric Alternating Attention module. Particularly, we transfer 2D attention priors from off-the-shelf foundation models to enhance 3D feature interaction and geometric consistency. Building upon FUSER, we further introduce FUSER-DF, an SE(3)$^N$ diffusion refinement framework to correct FUSER's estimates via denoising in the joint SE(3)$^N$ space. FUSER acts as a surrogate multiview registration model to construct the denoiser, and a prior-conditioned SE(3)$^N$ variational lower bound is derived for denoising supervision. Extensive experiments on 3DMatch, ScanNet and ArkitScenes demonstrate that our approach achieves the superior registration accuracy and outstanding computational efficiency.

Figures (6)

• Figure 1: Comparison of paradigms. Conventional multiview registration relies on redundant pairwise estimation (time-consuming and no global constraint) and pose synchronization (outlier sensitivity and high inductive bias). By contrast, our FUSER directly predicts global poses through unified feed-forward reasoning across all scans without any pairwise matching, delivering outstanding accuracy and efficiency (minutes$\rightarrow$seconds).
• Figure 2: Architecture of FUSER. It encodes unordered scans into a compact latent space via Absolute Geometric Encoding, then performs 2D attention prior-enhanced Geometric Alternating Attention for multiview reasoning and final pose regression.
• Figure 3: Pipeline of prior-aware SE(3)$^N$ denoising process. It integrates the prior pose estimates $(\hat{\mathbf{T}}_1,...,\hat{\mathbf{T}}_N)$ of FUSER into the denoising process, where FUSER, as the surrogate registration model, estimates the residual poses $(\hat{\mathbf{T}}_1^{t\rightarrow 0}, ..., \hat{\mathbf{T}}_N^{t\rightarrow 0})=\operatorname{\textcolor{rgb(166,35,35)}{FUSER}}(\mathcal{S}_t)$ to support progressive denoising over SE(3)$^N$ space.
• Figure 4: SE(3)$^N$ diffusion refinement in FUSER-DF visually refines FUSER’s pose estimation, yielding smoother surfaces.
• Figure 5: Qualitative comparison: FUSER surpasses SOTA GeoTrans qin2022geometric and PARENet yao2024pare descriptors with SGHR pose graph wang2023robust, achieving much higher accuracy and efficiency.