3D Reconstruction with Fast Dipole Sums

TL;DR

This work introduces a new point-based representation, the regularized dipole sum, which generalizes the winding number to allow for interpolation of per-point attributes in point clouds with noisy or outlier points.

Abstract

We introduce a method for high-quality 3D reconstruction from multi-view images. Our method uses a new point-based representation, the regularized dipole sum, which generalizes the winding number to allow for interpolation of per-point attributes in point clouds with noisy or outlier points. Using regularized dipole sums, we represent implicit geometry and radiance fields as per-point attributes of a dense point cloud, which we initialize from structure from motion. We additionally derive Barnes-Hut fast summation schemes for accelerated forward and adjoint dipole sum queries. These queries facilitate the use of ray tracing to efficiently and differentiably render images with our point-based representations, and thus update their point attributes to optimize scene geometry and appearance. We evaluate our method in inverse rendering applications against state-of-the-art alternatives, based on ray tracing of neural representations or rasterization of Gaussian point-based representations. Our method significantly improves 3D reconstruction quality and robustness at equal runtimes, while also supporting more general rendering methods such as shadow rays for direct illumination.

Figures (9)

• Figure 1: Overview of our method. During forward rendering (indicated by solid arrows), at each sample location along a ray, we interpolate geometry and appearance attributes from a point cloud through a fast primal dipole sum query. We pass appearance attributes through a shallow MLP to predict colors, and use geometry attributes to compute attenuation coefficients. We integrate along the ray to compute the rendered color and minimize the $L^1$-loss between the rendered and ground truth colors. During backpropagation (indicated by dashed arrows), we optimize geometry and appearance attributes of the point cloud through a fast adjoint dipole sum query.
• Figure 2: Using the original and regularized winding number fields on the unoptimized point cloud (left) for the BlendedMVS clock scene. The top row shows planar slices of the two fields: The original winding number is very noisy near point cloud locations due to the singular Poisson kernel, whereas the regularized winding number is much smoother. The insets visualize the singular and regularized kernels. The bottom row shows meshes extracted from the two fields using marching cubes: The original winding number results in strong artifacts, which the regularized winding number fixes.
• Figure 3: We visualize on the teaser scene geometry-related field quantities that we use for inverse rendering. From left to right: the initial geometry field with unit geometry attributes (equal to the regularized winding number in \ref{['eqn:regularized_pcwn']}), the optimized geometry field with learned geometry attributes (\ref{['eqn:geometry_field']}), the optimized vacancy field (\ref{['eqn:vacancy']}), and attenuation coefficients computed (\ref{['eqn:surface_density']}) along two different viewing directions.
• Figure 4: Our regularized dipole sum representation allows us to directly ray trace the optimized point cloud (where we use color to visualize normals, and size to visualize geometry attributes), achieving the same results as ray tracing a mesh without the need to extract one.
• Figure 5: Qualitative comparisons on the BlendedMVS (left) and DTU (right) datasets. The dashed circles indicate areas of interest. NeuS2 captures fine details, but produces noisy meshes with structural artifacts. Gaussian surfels produces floater artifacts that require manual filtering. By contrast, our method produces clean meshes with correct and artifact-free geometry. We provide interactive visualizations of results on the entire datasets on the project website.