Neural Fields in Robotics: A Survey

TL;DR

This survey maps Neural Fields (Occupancy, SDFs, NeRFs, and Gaussian Splatting) to robotics, detailing how continuous, differentiable 3D representations enable improved perception, planning, and control. It covers five application domains—pose estimation, manipulation, navigation, physics, and autonomous driving—highlighting representative methods, design choices, and practical trade-offs. Key contributions include a taxonomy of NF representations, performance trends, and open challenges such as real-time efficiency, dynamic scene handling, multi-sensor fusion, and sim-to-real transfer. The paper underscores the potential of integrating NF with foundation and diffusion models to advance robust, semantic, and open-vocabulary robotic systems, and points to future directions like physically grounded NF frameworks and standardized benchmarks.

Abstract

Neural Fields have emerged as a transformative approach for 3D scene representation in computer vision and robotics, enabling accurate inference of geometry, 3D semantics, and dynamics from posed 2D data. Leveraging differentiable rendering, Neural Fields encompass both continuous implicit and explicit neural representations enabling high-fidelity 3D reconstruction, integration of multi-modal sensor data, and generation of novel viewpoints. This survey explores their applications in robotics, emphasizing their potential to enhance perception, planning, and control. Their compactness, memory efficiency, and differentiability, along with seamless integration with foundation and generative models, make them ideal for real-time applications, improving robot adaptability and decision-making. This paper provides a thorough review of Neural Fields in robotics, categorizing applications across various domains and evaluating their strengths and limitations, based on over 200 papers. First, we present four key Neural Fields frameworks: Occupancy Networks, Signed Distance Fields, Neural Radiance Fields, and Gaussian Splatting. Second, we detail Neural Fields' applications in five major robotics domains: pose estimation, manipulation, navigation, physics, and autonomous driving, highlighting key works and discussing takeaways and open challenges. Finally, we outline the current limitations of Neural Fields in robotics and propose promising directions for future research. Project page: https://robonerf.github.io

Figures (20)

• Figure 1: Overview: This survey paper discusses a large variety of state-of-the-art Neural Field methods that enable robotics applications in pose estimation, manipulation, navigation, physics, and autonomous driving. Images adapted from hu2023nerfyen2020inerfli2024sgsshen2023F3RMlerftogo2023qureshi2024splatsimzeroshotsim2realtransferyu2024legschen2024saferxie2023physgaussianliu2024differentiablewu2023marsost2021neural.
• Figure 2: Growth of Neural Fields in Robotics: plotted as a rough number of publications vs. % of total neural field publications per year.
• Figure 3: Timeline of Neural Fields in Robotics paper showing key papers over the years divided into 5 major application areas.
• Figure 4: Neural Field Representations: Section \ref{['sec:theoretical_background']} discusses four core Neural Field representations --- Occupancy Networks mescheder2019occupancy, Signed Distance Fields park2019deepsdf, Neural Radiance Fields mildenhall2020nerf, and 3D Gaussian Splatting kerbl20233d.
• Figure 5: Taxonomy of selected key Neural Fields papers in five major robotics application areas.