R3-RECON: Radiance-Field-Free Active Reconstruction via Renderability

TL;DR

R3-RECON presents a radiance-fields-free approach to active reconstruction by learning a renderability field derived from online voxel statistics, enabling closed-form, millisecond NBV scoring without maintaining a radiance representation. The method decomposes renderability into directional bias, appearance stability, and resolution, computable via a fixed-size, memory-efficient voxel-statistics map, and augments this with a panoramic 360° utility extension for faster view-direction planning. Experiments on Replica-Dense show consistent improvements in novel-view quality and data efficiency over strong 3D Gaussian splat baselines while using substantially less GPU memory and compute. This renderability-centric framework offers a practical, plug-and-play alternative for real-time autonomous mapping and photorealistic rendering in resource-constrained platforms.

Abstract

In active reconstruction, an embodied agent must decide where to look next to efficiently acquire views that support high-quality novel-view rendering. Recent work on active view planning for neural rendering largely derives next-best-view (NBV) criteria by backpropagating through radiance fields or estimating information entropy over 3D Gaussian primitives. While effective, these strategies tightly couple view selection to heavy, representation-specific mechanisms and fail to account for the computational and resource constraints required for lightweight online deployment. In this paper, we revisit active reconstruction from a renderability-centric perspective. We propose $\mathbb{R}^{3}$-RECON, a radiance-fields-free active reconstruction framework that induces an implicit, pose-conditioned renderability field over SE(3) from a lightweight voxel map. Our formulation aggregates per-voxel online observation statistics into a unified scalar renderability score that is cheap to update and can be queried in closed form at arbitrary candidate viewpoints in milliseconds, without requiring gradients or radiance-field training. This renderability field is strongly correlated with image-space reconstruction error, naturally guiding NBV selection. We further introduce a panoramic extension that estimates omnidirectional (360$^\circ$) view utility to accelerate candidate evaluation. In the standard indoor Replica dataset, $\mathbb{R}^{3}$-RECON achieves more uniform novel-view quality and higher 3D Gaussian splatting (3DGS) reconstruction accuracy than recent active GS baselines with matched view and time budgets.

Figures (11)

• Figure 1: Overview of renderability. Left: green cameras denote training views and black dots denote novel test views; we visualize RGB renderings, per-view renderability, and panoramic renderability. Colored boxes highlight local regions whose novel-view artifacts vary with viewpoint and are well captured by renderability. Right: distributions over 227 novel views of LPIPS, SSIM, PSNR, and our renderability, showing strong alignment with image-space reconstruction quality.
• Figure 2: Renderability in IBR and radiance-field rendering. IBR estimates view feasibility from a coarse geometric proxy and source-view overlap, while 3DGS depends on how well primitive appearance is constrained across viewing directions.
• Figure 3: Renderability factors. Renderability combines directional bias, appearance noise, and a resolution term. $v$ denotes viewing direction and $z$ the observed radiance; orange points indicate the (unknown) radiance at a novel direction. Noise captures view-dependent variability, and bias measures how far the query direction deviates from the directional support of past observations. Extrapolation is detected by projecting past directions to the query tangent plane and checking whether the query lies outside their support region.
• Figure 4: System overview. Posed RGB-D updates a coarse occupancy grid for exploration and a fine voxel-statistics map for renderability. Candidate views are scored by exploration gain and renderability, with panoramic direction selection accelerated by sphere discretization and FoV aggregation.
• Figure 5: Evaluation pipeline. Each method runs online to generate a trajectory and collect posed RGB and scene pointcloud. A 3DGS model is then trained from the recorded data under matched budgets. For evaluation, test views are sampled from free-space grid locations, each rendered with six canonical viewing directions, to measure scene-level generalization on held-out novel views and reconstruction metrics.