Pow3R: Empowering Unconstrained 3D Reconstruction with Camera and Scene Priors

TL;DR

Pow3R addresses the need for a unified 3D regression model that can utilize test-time priors. It extends the DUSt3R foundation by conditioning the model on camera intrinsics, extrinsics, and depth maps via lightweight adapters within a transformer backbone. The model learns under random modality subsets during training, enabling robust performance when priors are available and preserving unconstrained RGB-only capabilities otherwise. The results show state-of-the-art performance across depth completion, multi-view depth, multi-view stereo, and multi-view pose tasks, and unlocks high-resolution and sparse-to-dense capabilities.

Abstract

We present Pow3r, a novel large 3D vision regression model that is highly versatile in the input modalities it accepts. Unlike previous feed-forward models that lack any mechanism to exploit known camera or scene priors at test time, Pow3r incorporates any combination of auxiliary information such as intrinsics, relative pose, dense or sparse depth, alongside input images, within a single network. Building upon the recent DUSt3R paradigm, a transformer-based architecture that leverages powerful pre-training, our lightweight and versatile conditioning acts as additional guidance for the network to predict more accurate estimates when auxiliary information is available. During training we feed the model with random subsets of modalities at each iteration, which enables the model to operate under different levels of known priors at test time. This in turn opens up new capabilities, such as performing inference in native image resolution, or point-cloud completion. Our experiments on 3D reconstruction, depth completion, multi-view depth prediction, multi-view stereo, and multi-view pose estimation tasks yield state-of-the-art results and confirm the effectiveness of Pow3r at exploiting all available information. The project webpage is https://europe.naverlabs.com/pow3r.

Figures (21)

• Figure 1: Qualitative comparison of raw 3D reconstructions between DUSt3R and Pow3R for the image pair on the left. DUSt3R cannot leverage auxiliary information potentially available at test time, resulting in noisy and low-resolution estimates, whereas Pow3R allows to inject such priors at test time in a versatile manner, enabling new capabilities such as high-resolution processing as a direct by-product.
• Figure 2: Overview of the model architecture. Following DUSt3R wang2024dust3r, images are encoded then decoded with a ViT backbone into pointmaps, from which focals, depthmaps and relative pose can be extracted. Pow3R introduces optional inputs to guide the regression with prior knowledge about the camera intrinsics and depth (fed into the encoder) and the pose (into the decoder).
• Figure 3: Architecture for injecting auxiliary modalities.Top: injection of optional intrinsics and depth into the encoder. Intrinsics are encoded into ray patches, sparse depth is patchified. Each of these modalities goes into a block-specific MLP and are token-wise added in the middle of the transformer block. Bottom: injection of optional relative pose into the decoder. The relative pose is fed to a first embedding layer followed by an MLP. This token is added to the CLS token of the decoder after the self-attention (SA) and cross-attention (CA), but before the MLP. Experiments show that injection in the first block only suffices.
• Figure 3: Multi-view depth evaluation: Pow3R with both pose and intrinsics performs better than DUSt3R. More comparisons with the state of the art can be found in the supplementary material. (Parentheses) denote training on data from the same domain.
• Figure 4: Native resolution example. Methods like DUSt3R require centered principal point and would thus downscale the image to its training resolution, leading to blurry prediction (top). With camera intrinsics input, we can process any crop in the image leading to native resolution prediction (bottom row), for example by dividing the image into 4 crops, that are independently process as the first input image, while the second input image of the network is set to the downscaled image.