Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MultiBARF: Integrating Imagery of Different Wavelength Regions by Using Neural Radiance Fields

Kana Kurata, Hitoshi Niigaki, Xiaojun Wu, Ryuichi Tanida

TL;DR

This work addresses the burden of multisensor data preparation by introducing MultiBARF, a dual-sensor extension of Neural Radiance Fields that shares one density channel across sensors while providing per-sensor color outputs. It eliminates explicit co-registration by leveraging BARF-style pose optimization and an alternating training scheme to learn a common 3D geometry and separate sensor appearances. The method demonstrates synthesis of paired visible and thermal images and corresponding depth from a single viewpoint, enabling practical multispectral fusion without calibrated cross-sensor geometry. The results indicate improved 3D reconstruction and accurate sensor-color registration, reducing data-preparation effort for multispectral sensing in real-world applications.

Abstract

Optical sensor applications have become popular through digital transformation. Linking observed data to real-world locations and combining different image sensors is essential to make the applications practical and efficient. However, data preparation to try different sensor combinations requires high sensing and image processing expertise. To make data preparation easier for users unfamiliar with sensing and image processing, we have developed MultiBARF. This method replaces the co-registration and geometric calibration by synthesizing pairs of two different sensor images and depth images at assigned viewpoints. Our method extends Bundle Adjusting Neural Radiance Fields(BARF), a deep neural network-based novel view synthesis method, for the two imagers. Through experiments on visible light and thermographic images, we demonstrate that our method superimposes two color channels of those sensor images on NeRF.

MultiBARF: Integrating Imagery of Different Wavelength Regions by Using Neural Radiance Fields

TL;DR

This work addresses the burden of multisensor data preparation by introducing MultiBARF, a dual-sensor extension of Neural Radiance Fields that shares one density channel across sensors while providing per-sensor color outputs. It eliminates explicit co-registration by leveraging BARF-style pose optimization and an alternating training scheme to learn a common 3D geometry and separate sensor appearances. The method demonstrates synthesis of paired visible and thermal images and corresponding depth from a single viewpoint, enabling practical multispectral fusion without calibrated cross-sensor geometry. The results indicate improved 3D reconstruction and accurate sensor-color registration, reducing data-preparation effort for multispectral sensing in real-world applications.

Abstract

Optical sensor applications have become popular through digital transformation. Linking observed data to real-world locations and combining different image sensors is essential to make the applications practical and efficient. However, data preparation to try different sensor combinations requires high sensing and image processing expertise. To make data preparation easier for users unfamiliar with sensing and image processing, we have developed MultiBARF. This method replaces the co-registration and geometric calibration by synthesizing pairs of two different sensor images and depth images at assigned viewpoints. Our method extends Bundle Adjusting Neural Radiance Fields(BARF), a deep neural network-based novel view synthesis method, for the two imagers. Through experiments on visible light and thermographic images, we demonstrate that our method superimposes two color channels of those sensor images on NeRF.

Paper Structure

This paper contains 10 sections, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Structure of MultiBARF
  5. Optimization of MultiBARF
  6. Experiments
  7. basic performance
  8. Practical performance
  9. Comparison of optimizing method
  10. Conclusion

Figures (7)

  • Figure 1: The original BARF structure.
  • Figure 2: Our MultiBARF structure.
  • Figure 3: The results with various balances of pre and post-branch layers. From the left, the number of layers of pre and post-branch are 4--6 and 8--2.
  • Figure 4: A part of the datasets; (a)"Warmer", (b)"Pack", (c)"Toy", (d)"Casserole", and (e)"Trace".
  • Figure 5: The synthesized results of the "Pack" dataset. The top and middle rows show the synthesized depth and color images by BARF trained with only visible light or thermal images, respectively. The bottom is the results of our MultiBARF, which learned both visible light and thermal images.
  • ...and 2 more figures