Online Adaptation for Implicit Object Tracking and Shape Reconstruction in the Wild

TL;DR

The paper tackles robust 3D object tracking and shape reconstruction from LiDAR video in cluttered, partially observable environments. It proposes online adaptation of a deep implicit representation (DeepSDF) to jointly track object poses and refine shapes across frames, leveraging a shape prior learned from ShapeNet. The joint objective minimizes pose, shape, and registration losses in an alternating optimization scheme, enabling the shape to guide tracking and vice versa. Experiments on Waymo and KITTI show state-of-the-art tracking performance and improved shape accuracy, demonstrating the practical value of incorporating video continuity and implicit priors in real-world autonomous perception.

Abstract

Tracking and reconstructing 3D objects from cluttered scenes are the key components for computer vision, robotics and autonomous driving systems. While recent progress in implicit function has shown encouraging results on high-quality 3D shape reconstruction, it is still very challenging to generalize to cluttered and partially observable LiDAR data. In this paper, we propose to leverage the continuity in video data. We introduce a novel and unified framework which utilizes a neural implicit function to simultaneously track and reconstruct 3D objects in the wild. Our approach adapts the DeepSDF model (i.e., an instantiation of the implicit function) in the video online, iteratively improving the shape reconstruction while in return improving the tracking, and vice versa. We experiment with both Waymo and KITTI datasets and show significant improvements over state-of-the-art methods for both tracking and shape reconstruction tasks. Our project page is at https://jianglongye.com/implicit-tracking .

Figures (8)

• Figure 1: 3D object tracking and shape reconstruction. We propose a novel framework to utilize DeepSDF to perform tracking and reconstruction simultaneously. We visualize five time steps of tracking and three reconstructed shapes. We obtain better and better shape close to the ground-truth point clouds over time, which in return helps the tracking.
• Figure 2: Overview of our method. After initialization of the shape code, tracking and shape adaptation are performed iteratively. At a specific frame, the incoming object point cloud is first aligned to the previous shape, and then the shape is adapted to the aligned point cloud. Both procedures are based on DeepSDF.
• Figure 3: Iterative optimization. We optimize pose by pushing the point cloud to the zero-level set of the SDF field in (a), and deform the SDF field to match the point cloud in (b) for better shape. We use different temperature of the color to represent the distances from the surface.
• Figure 4: Effectiveness of online adaptation mechanisms. Due to the online adaptation mechanism, both tracking(a) and reconstruction(b) performance could be boosted.
• Figure 5: Shape evolution on Waymo. In (a), from left to right is the results of shape reconstruction at different tracking steps. The ground-truth aggregated point clouds are shown in (b). The shapes are become more and more aligned with ground-truth point clouds via online adaptation.