iSDF: Real-Time Neural Signed Distance Fields for Robot Perception

TL;DR

iSDF introduces a real-time neural signed distance field reconstruction method that learns online from streaming depth data using continual learning and active replay. It employs a small MLP with positional encoding and a batch-distance bound-based self-supervision to produce accurate SDFs and gradients, enabling adaptive detail and denoising without pretraining. Across synthetic and real indoor datasets, iSDF outperforms voxel-grid baselines in SDF accuracy, collision-cost estimates, and gradient quality, while offering a memory-efficient, real-time solution suitable for navigation and manipulation planners. The work demonstrates the practical impact of neural SDFs for real-time robotics perception and planning, with ablations highlighting effective bound choices and compute efficiency.

Abstract

We present iSDF, a continual learning system for real-time signed distance field (SDF) reconstruction. Given a stream of posed depth images from a moving camera, it trains a randomly initialised neural network to map input 3D coordinate to approximate signed distance. The model is self-supervised by minimising a loss that bounds the predicted signed distance using the distance to the closest sampled point in a batch of query points that are actively sampled. In contrast to prior work based on voxel grids, our neural method is able to provide adaptive levels of detail with plausible filling in of partially observed regions and denoising of observations, all while having a more compact representation. In evaluations against alternative methods on real and synthetic datasets of indoor environments, we find that iSDF produces more accurate reconstructions, and better approximations of collision costs and gradients useful for downstream planners in domains from navigation to manipulation. Code and video results can be found at our project page: https://joeaortiz.github.io/iSDF/ .

Figures (14)

• Figure 1: We present iSDF, a system for real-time signed distance field reconstruction that optimises a randomly initialised network to regress to the signed distance for input 3D coordinates based on posed depth images from a live camera stream. To train the network in real-time we develop a batch-based self-supervision method. This network can be queried online to obtain collision costs and gradients for use by downstream planners in domains from navigation to manipulation. The trajectory (red) of the camera (green) is visualised on top of the zero level set (top row) and SDF slice (middle row) generated by iSDF over time (left to right).
• Figure 2: System diagram. iSDF's MLP regresses input 3D coordinates ${{\mathbf {x}}} = (x, y, z)$ to the signed distance $s = f_\theta({{\mathbf {x}}})$.
• Figure 3: Left: Different methods for computing the bound on the SDF prediction for sample point ${{\mathbf {x}}}$ (orange). Right: The bound for each method for sample points along a ray. Each plot shows a different ray and the distance along the ray is measured from the camera centre. The batch distance provides the tightest bounds on the true signed distance.
• Figure 4: Free space loss.$\mathcal{L_{\text{free\_space}}} ( f({{\mathbf {x}}}; \theta), b)$ for target SDF.
• Figure 5: SDF slices. Slices at constant height of the reconstructed SDF at the end of the sequence (we choose different heights for each sequence to capture the key scene elements). The meshes are shown for reference with the camera trajectory and keyframes selected by iSDF overlaid. For Voxblox and KinectFusion+, the slices are greyed out in the non-visible region as neither method makes predictions in this region. The ground truth ScanNet Dai:etal:CVPR2017 slices are greyed out in the non-visible regions as we only have ground truth SDF values in visible regions. White regions in the Voxblox and KinectFusion+ slices are regions that are visible but unmapped (i.e. no rays reach this region).