Blurred LiDAR for Sharper 3D: Robust Handheld 3D Scanning with Diffuse LiDAR and RGB

TL;DR

The paper tackles robust 3D reconstruction with handheld devices in challenging low-texture, low-light, and low-albedo environments by fusing diffuse LiDAR with RGB. It introduces Gaussian surfel-based rendering and a scene-adaptive loss to resolve depth ambiguities inherent to diffuse LiDAR’s wide-field measurements, leveraging RGB cues for disambiguation. Through recoverability analysis, simulated tests, and real-world experiments, the approach demonstrates that diffuse LiDAR can outperform traditional sparse LiDAR in limited-view conditions, yielding accurate color and geometry. This work enables practical, robust mobile 3D scanning using commodity sensors, with potential impact on AR/VR, robotics, and mobile capture workflows.

Abstract

3D surface reconstruction is essential across applications of virtual reality, robotics, and mobile scanning. However, RGB-based reconstruction often fails in low-texture, low-light, and low-albedo scenes. Handheld LiDARs, now common on mobile devices, aim to address these challenges by capturing depth information from time-of-flight measurements of a coarse grid of projected dots. Yet, these sparse LiDARs struggle with scene coverage on limited input views, leaving large gaps in depth information. In this work, we propose using an alternative class of "blurred" LiDAR that emits a diffuse flash, greatly improving scene coverage but introducing spatial ambiguity from mixed time-of-flight measurements across a wide field of view. To handle these ambiguities, we propose leveraging the complementary strengths of diffuse LiDAR with RGB. We introduce a Gaussian surfel-based rendering framework with a scene-adaptive loss function that dynamically balances RGB and diffuse LiDAR signals. We demonstrate that, surprisingly, diffuse LiDAR can outperform traditional sparse LiDAR, enabling robust 3D scanning with accurate color and geometry estimation in challenging environments.

Figures (7)

• Figure 1: Blurred LiDAR for Sharper 3D: We propose leveraging (a) the complementary strengths of RGB with a diffuse (blurred) LiDAR for robust handheld scanning (b) on limited views in challenging low-texture, low-light, low-albedo scenes. Our method dynamically balances RGB and diffuse LiDAR sensor inputs to estimate (c) precise color, depth, and normals, from which we can (d) reconstruct accurate color-3D meshes. Our approach leveraging blurred LiDAR, counterintuitively, can (e) improve 3D reconstruction over conventional LiDAR, which uses sparse spot illumination.
• Figure 2: Diffuse LiDAR + RGB for surface reconstruction. (a) Sparse (conventional) LiDAR vs Diffuse LiDAR. Sparse LiDAR projects a grid of points which enable precise timing returns corresponding to individual depths; alternatively, diffuse LiDAR projects a diffuse flash illumination and measures the returns over a wide per-pixel instantaneous field-of-view (IFOV), increasing spatial coverage but also ambiguity in inferred depth. (b) Diffuse LiDAR and RGB have complementary strengths; RGB provides dense spatial and color information, while diffuse LiDAR provides coarse, metric depth even under challenging scenarios.
• Figure 3: Improved recoverability with diffuse LiDAR when input views are limited. Full rank is 900 in our analysis simulation. Diffuse LiDAR has greater voxel coverage than conventional sparse LiDAR; this greater coverage can improve rank, and thereby recoverability, when using a limited number of input views. We consider this specific limited-view domain in this work. As the number of input views increases, sparse LiDAR can eventually provide sufficient coverage for scene recoverability.
• Figure 4: Inverse rendering with RGB and diffuse LiDAR. We consider a compact hardware setting with co-located RGB camera and diffuse LiDAR. At each view, we capture an (a) RGB image and coarse ($8 \times 8$) histograms, for which each pixel contains mixed signal from a wide IFOV $\omega$. We perform (b) analysis-by-synthesis reconstruction using Gaussian surfels, sampling rays within each pixel IFOV and rendering transients with alpha-weighted differentiable binning. Loss signal from RGB and transient inputs are balanced dynamically with a scene-adaptive loss producing (c) high-fidelity RGB and depth/normals for accurate mesh reconstruction.
• Figure 5: Qualitative comparisons on rendered scenes with varying texture. We enable (a) accurate RGB novel-view synthesis on full-texture scenes, where RGB may be prioritized by our adaptive loss. When (b) textured objects are on textureless planes, we enable greater object-plane separation and peripheral region robustness; on textureless objects, we (c) enhance geometry estimation within the object convex hull visible from RGB. We also (d) improve over sparse LiDAR in scenes without RGB signal, where our adaptive loss may weight diffuse LiDAR more heavily. We enable (e) accurate color-mesh estimation across a wide range of texture and object variations.