NeRFs in Robotics: A Survey

TL;DR

This survey provides a structured overview of Neural Radiance Fields (NeRFs) in robotics, detailing how NeRFs enable rich 3D scene understanding and interaction while highlighting challenges in scalability, real-time performance, and multi-modal perception. It divides the discussion into applications—covering reconstruction, segmentation/editing, navigation, and manipulation—and advances—focusing on realism, efficiency, and adaptability to large-scale and unseen environments. Key contributions include taxonomy of NeRF-based robotic methods, evaluation metrics across tasks, and forward-looking directions such as multi-robot map fusion, robust relocalization, sim-to-real rendering, and physics-informed generalization. The paper emphasizes the shift from static view synthesis to integrated perception-action systems that leverage hybrid representations and self-supervised learning to enable robust, real-time robotic operation in dynamic environments. This work thus delineates a roadmap for deploying NeRFs in real-world robotics, outlining both practical methods and open research opportunities.

Abstract

Detailed and realistic 3D environment representations have been a long-standing goal in the fields of computer vision and robotics. The recent emergence of neural implicit representations has introduced significant advances to these domains, enabling numerous novel capabilities. Among these, Neural Radiance Fields (NeRFs) have gained considerable attention because of their considerable representational advantages, such as simplified mathematical models, low memory footprint, and continuous scene representations. In addition to computer vision, NeRFs have demonstrated significant potential in robotics. Thus, we present this survey to provide a comprehensive understanding of NeRFs in the field of robotics. By exploring the advantages and limitations of NeRF as well as its current applications and future potential, we aim to provide an overview of this promising area of research. Our survey is divided into two main sections: \textit{Applications of NeRFs in Robotics} and \textit{Advances for NeRFs in Robotics}, from the perspective of how NeRF enters the field of robotics. In the first section, we introduce and analyze some works that have been or could be used in robotics for perception and interaction tasks. In the second section, we show some works related to improving NeRF's own properties, which are essential for deploying NeRFs in robotics. In the discussion section of the review, we summarize the existing challenges and provide valuable future research directions.

Figures (19)

• Figure 1: A taxonomy of NeRFs in robotics.
• Figure 2: The training process of NeRF. The image is sourced from mildenhall2020nerf. For each viewpoint, NeRF assumes a ray along the direction connecting the camera origin and a pixel of the target image. Multiple points are sampled along this ray in the reconstructed scene. The 5D coordinates of these points (3D position $+$ 2D orientation) are input into an MLP, which outputs their corresponding colour and density values. Next, the volume rendering is performed by integrating the colour and density of sampled points along a ray, producing the estimated colour of the target pixel. Finally, the difference between the estimated colour and the ground truth is used to update the entire network through the rendering loss. The NeRF network is trained through this iterative process.
• Figure 3: Chronological: NeRFs for Scene Reconstruction in Section \ref{['reconstruction']}.
• Figure 4: An illustration of NeRF for static reconstruction. Fig. \ref{['Indoor_Static_Reconstruction']} and Fig. \ref{['Outdoor_Static_Reconstruction']} are originally shown in zhu2022nice and deng2023nerf, respectively.
• Figure 5: An illustration of NeRF for dynamic reconstruction. Fig. \ref{['Deformation-based']} and Fig. \ref{['Flow-based']} are originally shown in tretschk2021non and li2021neural, respectively.