OpticFusion: Multi-Modal Neural Implicit 3D Reconstruction of Microstructures by Fusing White Light Interferometry and Optical Microscopy

TL;DR

Problem: WLI provides precise 3D topography but lacks natural color texture, limiting microscale analysis. Approach: OpticFusion fuses multi-view WLI and OM via a two-step data association to place both modalities into a common absolute-scale frame and trains a neural implicit SDF with color decomposition to yield view-independent color textures. Contributions: first microscale textured reconstruction using only a commercial WLI and an OM, a two-step pose estimation, a three-network color model with a residual component, and evaluation on real and synthetic datasets including roughness analysis. Findings: OpticFusion achieves higher-fidelity geometry and natural textures compared with conventional Poisson/MVS baselines and OM-only NN methods, and its synthetic experiments quantify multimodal benefits. Impact: enables practical microscale analysis across disciplines and provides public code and dataset.

Abstract

White Light Interferometry (WLI) is a precise optical tool for measuring the 3D topography of microstructures. However, conventional WLI cannot capture the natural color of a sample's surface, which is essential for many microscale research applications that require both 3D geometry and color information. Previous methods have attempted to overcome this limitation by modifying WLI hardware and analysis software, but these solutions are often costly. In this work, we address this challenge from a computer vision multi-modal reconstruction perspective for the first time. We introduce OpticFusion, a novel approach that uses an additional digital optical microscope (OM) to achieve 3D reconstruction with natural color textures using multi-view WLI and OM images. Our method employs a two-step data association process to obtain the poses of WLI and OM data. By leveraging the neural implicit representation, we fuse multi-modal data and apply color decomposition technology to extract the sample's natural color. Tested on our multi-modal dataset of various microscale samples, OpticFusion achieves detailed 3D reconstructions with color textures. Our method provides an effective tool for practical applications across numerous microscale research fields. The source code and our real-world dataset are available at https://github.com/zju3dv/OpticFusion.

Figures (12)

• Figure 1: Multi-modal neural implicit reconstruction of microscale samples with multi-view images from white light interferometer and optical microscope.(a) The principle of multi-view WLI scanning. (b, c) The output of white light interferometer and optical microscope. (d) The multi-view imaging with an optical microscope. (e) OpticFusion reconstructs 3D models of microstructures with their natural surface colors, without costly modifications of the WLI system.
• Figure 2: System pipeline of OpticFusion. Our method takes multi-view WLI and OM images of a microscale sample as input. Through a two-step data association process, we compute the pose of each WLI and OM image in the same absolute-scale coordinate system. Using differentiable volume rendering techniques, we train the neural implicit representation of the microstructure's geometry and view-independent color under the supervision of the multi-modal data. The result is a textured 3D surface model of the microscale sample.
• Figure 3: Qualitative reconstruction results on our WLI-OM real-world dataset. Compared with other methods, our reconstruction results have more accurate surface geometry and natural texture colors. Please see more visualizations in the supplementary material.
• Figure 4: Roughness measurement of microscale surfaces. We calculate the roughness of two surfaces and obtain cross-sections in two directions.
• Figure 5: Surface detail analysis of a biological sample. We isolate separate 3D models of a single scale and a tiny burr on the butterfly's wing. Additionally, we provide cross-sections of multiple continuous scales and veins.