Magic NeRF Lens: Interactive Fusion of Neural Radiance Fields for Virtual Facility Inspection

TL;DR

A data fusion technique is introduced to merge a NeRF model with the polygonal representation of it’s corresponding CAD model, which optimizes VR NeRF rendering through magic-lens-style interactions while introducing a novel industrial visualization design that can support practical tasks such as facility maintenance planning and redesign.

Abstract

Large industrial facilities such as particle accelerators and nuclear power plants are critical infrastructures for scientific research and industrial processes. These facilities are complex systems that not only require regular maintenance and upgrades but are often inaccessible to humans due to various safety hazards. Therefore, a virtual reality (VR) system that can quickly replicate real-world remote environments to provide users with a high level of spatial and situational awareness is crucial for facility maintenance planning. However, the exact 3D shapes of these facilities are often too complex to be accurately modeled with geometric primitives through the traditional rasterization pipeline. In this work, we develop Magic NeRF Lens, an interactive framework to support facility inspection in immersive VR using neural radiance fields (NeRF) and volumetric rendering. We introduce a novel data fusion approach that combines the complementary strengths of volumetric rendering and geometric rasterization, allowing a NeRF model to be merged with other conventional 3D data, such as a computer-aided design model. We develop two novel 3D magic lens effects to optimize NeRF rendering by exploiting the properties of human vision and context-aware visualization. We demonstrate the high usability of our framework and methods through a technical benchmark, a visual search user study, and expert reviews. In addition, the source code of our VR NeRF framework is made publicly available for future research and development.

Figures (7)

• Figure 1: Screenshots from the Magic NeRF Lens framework: (a) illustrates the data fusion of a high-resolution NeRF rendering and corresponding CAD model through mixed reality tunneling 19-MR-Tunneling, and (b) illustrates the 3D NeRF drawing interaction using the CAD model as context.
• Figure 2: (a): Schematic sketch of the design of our interactive lens effect where the FoV of the NeRF camera is reduced and the actual NeRF render frustum is defined as a box rather than a pyramid to reduce NeRF render load. (b): Screenshot of the magic lens effect, where the blue box visualizes the NeRF crop box that is dynamically following the user's head movement.
• Figure 3: Illustration of our system extension to immersive-ngp. (a-b): NeRF model manipulation, where users can rotate, scale, and translate the model in VR. (c-d): NeRF model crop box manipulation, where users can rotate and scale the render volume crop box in VR. (e-f): Volume editing, where users could dynamically delete and remove an area of the render volume via 3D VR drawing.
• Figure 4: Overview of the data fusion pipeline to merge a NeRF model with its corresponding CAD model, illustrating sub-processes from image pre-processing, NeRF model training, scene cleaning, scene alignment, scene merging, and examples of final integration and interaction using different features of our framework.
• Figure 5: Systematic benchmark results showing (left): the relationship between rendering FoV and average frame time following a predefined fixed path without a VR HMD attached, and (right): the trend between rendering FoV and average frame timing for a test user following only approximately the same 3D trace. (R) indicates reduced rendering via 3D NeRF drawing.