Auto3R: Automated 3D Reconstruction and Scanning via Data-driven Uncertainty Quantification

TL;DR

Auto3R addresses fully automated 3D scanning by learning data-driven uncertainty maps that jointly model 2D appearance and 3D geometry. It uses depth-aware blending and depth-uncertainty reweighting within a 3D Gaussian Splatting framework to guide view selection, and extends to path-level scanning for robotic platforms. The approach achieves state-of-the-art reconstruction quality on object and scene benchmarks, including specular objects, and demonstrates real-world deployment on a robot. This work enables efficient, automated digitization of complex scenes and materials with practical implications for robotics and digital asset creation.

Abstract

Traditional high-quality 3D scanning and reconstruction typically relies on human labor to plan the scanning procedure. With the rapid development of embodied systems such as drones and robots, there is a growing demand of performing accurate 3D scanning and reconstruction in an fully automated manner. We introduce Auto3R, a data-driven uncertainty quantification model that is designed to automate the 3D scanning and reconstruction of scenes and objects, including objects with non-lambertian and specular materials. Specifically, in a process of iterative 3D reconstruction and scanning, Auto3R can make efficient and accurate prediction of uncertainty distribution over potential scanning viewpoints, without knowing the ground truth geometry and appearance. Through extensive experiments, Auto3R achieves superior performance that outperforms the state-of-the-art methods by a large margin. We also deploy Auto3R on a robot arm equipped with a camera and demonstrate that Auto3R can be used to effectively digitize real-world 3D objects and delivers ready-to-use and photorealistic digital assets. Our homepage: https://tomatoma00.github.io/auto3r.github.io .

Figures (16)

• Figure 1: Left: The scanning viewpoints selected by Auto3R, exhibiting a tendency to converge toward areas with occlusion. Right: Auto3R achieves accurate reconstruction quality and significantly outperforms the state-of-the-art methods.
• Figure 2: Left: Reconstruction from randomly sampled viewpoints can lead to incomplete observation (highlighted with red surface) and thus low-quality results. Right: Active planning formulated with uncertainty prediction solves the problem by selectively chose observation viewpoints for scanning.
• Figure 3: An illustration of our automated scanning and reconstruction methods. Given some images from scanning views, we reconstruction them then render the image and depth map on some candidate scanning viewpoints. We proposed an uncertainty quantification on them, obtaining the uncertainty of each candidate viewpoint. Finally scan on the most uncertain viewpoints, and repeat the process.
• Figure 4: Training pipeline of our data-driven uncertainty map network. We render 3000 Objaverse objects with random illumination, train 3DGS using 12–25 sparse views, and render novel views for training the model. The uncertainty is then formulated as the SSIM between the rendered and ground-truth novel views.
• Figure 5: Illustration of our uncertainty quantification. Rendered-image uncertainty is integrated along depth using perspective projection, where deeper pixels receive higher weights, and is further reweighted by depth-map uncertainty.