Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Novel View Synthesis with Neural Radiance Fields for Industrial Robot Applications

Markus Hillemann, Robert Langendörfer, Max Heiken, Max Mehltretter, Andreas Schenk, Martin Weinmann, Stefan Hinz, Christian Heipke, Markus Ulrich

TL;DR

This work tackles the robustness and practicality of neural radiance fields (NeRFs) for industrial robotics by replacing Structure‑from‑Motion with robot‑driven, metric‑scale camera poses derived from hand–eye calibration. Through a multi‑dataset study, it compares Nerfacto and 3DGS under COLMAP and robot pose regimes, finding that 3DGS consistently delivers superior image quality and that online pose refinement can degrade performance in industrial settings. An ensemble‑based uncertainty approach is proposed to gauge the quality of synthesized views, revealing both the potential and limitations of ensembles for in‑distribution and out‑of‑distribution scenarios. Overall, robot‑assisted NeRF pipelines prove to be more robust and faster in industrial contexts with challenging textures and reflections, and the work outlines directions for extending NeRFs to reconstruction, thermal imaging, and cross‑spectral analysis.

Abstract

Neural Radiance Fields (NeRFs) have become a rapidly growing research field with the potential to revolutionize typical photogrammetric workflows, such as those used for 3D scene reconstruction. As input, NeRFs require multi-view images with corresponding camera poses as well as the interior orientation. In the typical NeRF workflow, the camera poses and the interior orientation are estimated in advance with Structure from Motion (SfM). But the quality of the resulting novel views, which depends on different parameters such as the number and distribution of available images, as well as the accuracy of the related camera poses and interior orientation, is difficult to predict. In addition, SfM is a time-consuming pre-processing step, and its quality strongly depends on the image content. Furthermore, the undefined scaling factor of SfM hinders subsequent steps in which metric information is required. In this paper, we evaluate the potential of NeRFs for industrial robot applications. We propose an alternative to SfM pre-processing: we capture the input images with a calibrated camera that is attached to the end effector of an industrial robot and determine accurate camera poses with metric scale based on the robot kinematics. We then investigate the quality of the novel views by comparing them to ground truth, and by computing an internal quality measure based on ensemble methods. For evaluation purposes, we acquire multiple datasets that pose challenges for reconstruction typical of industrial applications, like reflective objects, poor texture, and fine structures. We show that the robot-based pose determination reaches similar accuracy as SfM in non-demanding cases, while having clear advantages in more challenging scenarios. Finally, we present first results of applying the ensemble method to estimate the quality of the synthetic novel view in the absence of a ground truth.

Novel View Synthesis with Neural Radiance Fields for Industrial Robot Applications

TL;DR

This work tackles the robustness and practicality of neural radiance fields (NeRFs) for industrial robotics by replacing Structure‑from‑Motion with robot‑driven, metric‑scale camera poses derived from hand–eye calibration. Through a multi‑dataset study, it compares Nerfacto and 3DGS under COLMAP and robot pose regimes, finding that 3DGS consistently delivers superior image quality and that online pose refinement can degrade performance in industrial settings. An ensemble‑based uncertainty approach is proposed to gauge the quality of synthesized views, revealing both the potential and limitations of ensembles for in‑distribution and out‑of‑distribution scenarios. Overall, robot‑assisted NeRF pipelines prove to be more robust and faster in industrial contexts with challenging textures and reflections, and the work outlines directions for extending NeRFs to reconstruction, thermal imaging, and cross‑spectral analysis.

Abstract

Neural Radiance Fields (NeRFs) have become a rapidly growing research field with the potential to revolutionize typical photogrammetric workflows, such as those used for 3D scene reconstruction. As input, NeRFs require multi-view images with corresponding camera poses as well as the interior orientation. In the typical NeRF workflow, the camera poses and the interior orientation are estimated in advance with Structure from Motion (SfM). But the quality of the resulting novel views, which depends on different parameters such as the number and distribution of available images, as well as the accuracy of the related camera poses and interior orientation, is difficult to predict. In addition, SfM is a time-consuming pre-processing step, and its quality strongly depends on the image content. Furthermore, the undefined scaling factor of SfM hinders subsequent steps in which metric information is required. In this paper, we evaluate the potential of NeRFs for industrial robot applications. We propose an alternative to SfM pre-processing: we capture the input images with a calibrated camera that is attached to the end effector of an industrial robot and determine accurate camera poses with metric scale based on the robot kinematics. We then investigate the quality of the novel views by comparing them to ground truth, and by computing an internal quality measure based on ensemble methods. For evaluation purposes, we acquire multiple datasets that pose challenges for reconstruction typical of industrial applications, like reflective objects, poor texture, and fine structures. We show that the robot-based pose determination reaches similar accuracy as SfM in non-demanding cases, while having clear advantages in more challenging scenarios. Finally, we present first results of applying the ensemble method to estimate the quality of the synthetic novel view in the absence of a ground truth.
Paper Structure (13 sections, 9 equations, 8 figures, 2 tables)

This paper contains 13 sections, 9 equations, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Setup of the industrial robot application
  4. Experimental results
  5. Datasets
  6. Pose accuracy
  7. Evaluation of the NeRF approaches
  8. Quantitative results of the NeRF methods
  9. Qualitative results of the NeRF methods
  10. Uncertainty quantification with a NeRF ensemble
  11. In-distribution experiment.
  12. Out-of-distribution experiment.
  13. Conclusion

Figures (8)

  • Figure 1: (a) Hardware setup and data capturing. (b) Relevant 3D coordinate systems, observed poses (green), and poses calculated during hand--eye calibration (red); adapted from ulrich2024uncertainty.
  • Figure 2: Example image of each dataset.
  • Figure 3: Result of COLMAP for Dataset 4, i.e., the reconstructed sparse point cloud and the estimated poses.
  • Figure 4: Reference view and rendered result of the robot version of 3DGS for an image of dataset 3.
  • Figure 5: Reference view and rendered result of the COLMAP version of Nerfacto-big without pose refinement for an image of dataset 4.
  • ...and 3 more figures