FNIN: A Fourier Neural Operator-based Numerical Integration Network for Surface-form-gradients

TL;DR

This work proposes a Fourier neural operator-based Numerical Integration Network (FNIN) within a two-stage optimization framework, harnessing an advanced Fourier neural operator to approximate the solution operator in Fourier space within a two-stage optimization framework.

Abstract

Surface-from-gradients (SfG) aims to recover a three-dimensional (3D) surface from its gradients. Traditional methods encounter significant challenges in achieving high accuracy and handling high-resolution inputs, particularly facing the complex nature of discontinuities and the inefficiencies associated with large-scale linear solvers. Although recent advances in deep learning, such as photometric stereo, have enhanced normal estimation accuracy, they do not fully address the intricacies of gradient-based surface reconstruction. To overcome these limitations, we propose a Fourier neural operator-based Numerical Integration Network (FNIN) within a two-stage optimization framework. In the first stage, our approach employs an iterative architecture for numerical integration, harnessing an advanced Fourier neural operator to approximate the solution operator in Fourier space. Additionally, a self-learning attention mechanism is incorporated to effectively detect and handle discontinuities. In the second stage, we refine the surface reconstruction by formulating a weighted least squares problem, addressing the identified discontinuities rationally. Extensive experiments demonstrate that our method achieves significant improvements in both accuracy and efficiency compared to current state-of-the-art solvers. This is particularly evident in handling high-resolution images with complex data, achieving errors of fewer than 0.1 mm on tested objects.

Figures (12)

• Figure 1: Reconstruction of the "HOUSE" in the LUCES luces dataset. Previous Rec-Net lichy introduces sharp features and distortion at discontinuities, while our FNIN preserves the desired details.
• Figure 2: The Fourier neural operator-based numerical integration network operates within a two-stage framework. In stage I, the input normals are downsampled to different resolutions. In this iterative structure, the output depth is initially set by the initial network (blue) and subsequently refined by the iterative network (green). At each resolution, the Fourier backbone receives the upsampled output from the previous resolution and performs integration under the supervision of a detail weighted loss to preserve discontinuities. In stage II, the result is refined through a one-step weighted least squares optimization.
• Figure 3: Integration is performed at an example resolution. Gradient g for the input is computed using the normal and the upsampled output from the previous resolution with Eq. \ref{['equation2']}. The integration is then carried out within NO framework, with the solution approximated in Fourier space. Finally, the relative weight for discontinuities is estimated by the attention network.
• Figure 4: A simple example of (a) our assumption and (b) original natural boundary condition. Our assumption on given approximation green point C ensures the uniqueness otherwise the solution is ambiguous because value at other coordinates (blue point B) is also unknown.
• Figure 5: Qualitative results on "BRAR", "COW" and "POT 1" objects from DiLiGenT diligent Dataset. The black numbers under the error maps indicate the MAE (mm). The best result for each object is highlighted in bold.