Towards Fast and Scalable Normal Integration using Continuous Components

TL;DR

This work addresses the scalability bottleneck of surface normal integration by recasting the problem as estimating relative scales across continuous surface components. Each component is independently reconstructed, and a global, inter-component optimization aligns these patches through relative scales, with an optional merging step to further reduce problem size. The approach yields state-of-the-art accuracy on standard benchmarks like DiLiGenT and demonstrates orders-of-magnitude speedups on high-resolution normal maps, while preserving discontinuities. The method combines a simple, effective component-formation heuristic with robust relative-scale optimization and outlier reweighting, offering a practical pathway to scalable, discontinuity-preserving normal integration. The authors also provide open-source code to encourage adoption and further research.

Abstract

Surface normal integration is a fundamental problem in computer vision, dealing with the objective of reconstructing a surface from its corresponding normal map. Existing approaches require an iterative global optimization to jointly estimate the depth of each pixel, which scales poorly to larger normal maps. In this paper, we address this problem by recasting normal integration as the estimation of relative scales of continuous components. By constraining pixels belonging to the same component to jointly vary their scale, we drastically reduce the number of optimization variables. Our framework includes a heuristic to accurately estimate continuous components from the start, a strategy to rebalance optimization terms, and a technique to iteratively merge components to further reduce the size of the problem. Our method achieves state-of-the-art results on the standard normal integration benchmark in as little as a few seconds and achieves one-order-of-magnitude speedup over pixel-level approaches on large-resolution normal maps.

Figures (9)

• Figure 1: Example of surface discontinuities.Left: The chair in the foreground is separated by a full surface discontinuity from the wall in the background. Infinitely many depth maps can describe the input normal map, and solutions of the integration are up to the relative scales of the chair and the wall. Right: The surface points $\boldsymbol{p_a}$ and $\boldsymbol{p_b}$ are separated by a discontinuity along the path in red, but the yellow path connects them along the surface without discontinuities. Source of object meshes : MeshOfficePropsLowpoly2021 (left), Shi2016DiLiGenT (right).
• Figure 2: Method overview. We recast normal estimation as the estimation of the relative scale of continuous components. (a) Form continuous components based on the similarity of normals at neighboring pixels, independently reconstruct associated surface patches via per-component optimization (\ref{['sec:method_formation_filling_initial_components']}). (b) Align surface patches and recover discontinuities using relative scale optimization based on inter-component constraints (\ref{['sec:method_relative_scale_optimization']}). (c) Optionally, during optimization, merge components to further reduce complexity (\ref{['sec:method_optional_component_merging']}).
• Figure 3: Graph-theory interpretation of our framework.(a) Pixel-level optimization can be seen as operating on a dense graph $\mathcal{G}_0=(V, E)$. (b) We propose partitioning the per-pixel vertices $V$ into components. (c) The resulting optimization operates on a quotient graph$\mathcal{Q}_0$\ref{['eq:quotient_graph_definition']}. (d) Optionally, components can be merged into one another, forming a reduced quotient graph (\ref{['sec:method_optional_component_merging']}).
• Figure 4: Evaluation on large-scale normal maps. All experiments use $\Delta E_{\mathrm{max}}=10^{-3}$, $T=15$, outlier reweighting, and $\mathrm{freq}_{\mathrm{merging}}=5$ (where applicable). Total execution time $t$ and, where ground-truth reconstruction is available, mean relative depth error (RE) are reported. $\mathrm{M}$ and $\mathrm{NM}$ denote our version with and without merging, respectively. Sources of the normal maps: col. $1$-$2$: rendered from meshes MeshCozyModernBedroom2023, MeshCozyLivingRoomBaked2022; col. $3$-$4$: from photometric stereo Ikehata2023SDMUniPS, data from Ikehata2023SDMUniPS; col. $5$-$6$: predicted by DSINE Bae2024DSINE on in-the-wild images Bae2024DSINE, ImageWinterWeddingCake2007.
• Figure 5: Outlier reweighting \ref{['eq:outlier_reweighting']} as a function of $\log_{10}(|\overline{\chi}^{(t-1)}_{b\rightarrow a}|)$, for$L=10^{-3}$ and $U=10^{-5}$.