NeRF-To-Real Tester: Neural Radiance Fields as Test Image Generators for Vision of Autonomous Systems

TL;DR

This work tackles the simulation-to-reality gap in autonomous infrastructure inspection by leveraging Neural Radiance Fields (NeRFs) to generate realistic and diverse test images for vision components. It introduces N2R-Tester, a metamorphic testing framework that reconstructs NeRF scenes from 2D data, renders original and transformed viewpoints, and measures inconsistencies in SUT outputs under pose perturbations. The experimental evaluation across eight vision components in AUVs and UAVs demonstrates that NeRF-based test data can reveal domain-shift effects and detect reliability issues more effectively than traditional image mutations, while also quantifying realism and rendering efficiency. The approach offers a practical, scalable means to stress-test perception systems without exhaustive real-world data collection, with potential extensions to stereo testing and dynamic scenes.

Abstract

Autonomous inspection of infrastructure on land and in water is a quickly growing market, with applications including surveying constructions, monitoring plants, and tracking environmental changes in on- and off-shore wind energy farms. For Autonomous Underwater Vehicles and Unmanned Aerial Vehicles overfitting of controllers to simulation conditions fundamentally leads to poor performance in the operation environment. There is a pressing need for more diverse and realistic test data that accurately represents the challenges faced by these systems. We address the challenge of generating perception test data for autonomous systems by leveraging Neural Radiance Fields to generate realistic and diverse test images, and integrating them into a metamorphic testing framework for vision components such as vSLAM and object detection. Our tool, N2R-Tester, allows training models of custom scenes and rendering test images from perturbed positions. An experimental evaluation of N2R-Tester on eight different vision components in AUVs and UAVs demonstrates the efficacy and versatility of the approach.

Figures (11)

• Figure 1: Trade-offs between the data realness and amount/cost.
• Figure 2: (Top) We take a set of real images of a scene $i_\textrm{real} \!\in\! S$ as input. The NeRF model $\theta$ learns to map the estimated camera poses $\mathbf{x}, \mathbf{d}$ (yellow) to colour $\mathbf{c}$ and volumetric density $\sigma$. (Bottom) We pass new camera poses $(\mathbf{x}, \mathbf{d})$ to the NeRF model $\theta$, creating previously unseen views $i_\textrm{nerf}$ of the scene. Camera rays $\mathbf{r}(t)$ (red) are projected from the camera origin along the trajectory between a near and far bounds $t_n$, $t_f$.
• Figure 3: Metamorphic testing: Metamorphic relations (MRs) are established between pairs of test inputs and between the corresponding outputs. The test fails if MR2 does not hold.
• Figure 4: N2R-Tester: (1) an image processing SUT $f$, (2) a metamorphic relation (MR) in form of a pose transformation $\tau$ rendered in a NeRF model, (3) an MR $\delta$ checking inconsistencies between the SUT outputs $f(i_\textrm{nerf})$ and $f(i_\tau)$.
• Figure 5: DeepXplore mutations of real images of a dozer, with (a) changes in brightness, (b) a random pixel patch and (c) multiple black patches, imitating camera failure modes.