Physics-Guided Fusion for Robust 3D Tracking of Fast Moving Small Objects

TL;DR

This work tackles robust 3D tracking of fast-moving small objects by integrating a deep-learning detector with a physics-guided 3D tracking framework that leverages depth information from RGB-D sensors. By modeling kinematics and collision dynamics, the proposed system maintains accurate trajectories through occlusions and rapid direction changes, outperforming state-of-the-art DL trackers by substantial margins (ADE/AMD reductions up to 63–90% and 65–80%, respectively) on a racquetball-focused RGB-D dataset. The approach combines precise 3D pose estimation, depth correction, and an outlier detection/correction mechanism to enhance robustness and enable real-time operation on embedded hardware (NVIDIA Jetson TX2). Overall, the paper demonstrates the practical value of physics-informed tracking for high-speed, small-object applications in robotics, sports analytics, and autonomous perception.

Abstract

While computer vision has advanced considerably for general object detection and tracking, the specific problem of fast-moving tiny objects remains underexplored. This paper addresses the significant challenge of detecting and tracking rapidly moving small objects using an RGB-D camera. Our novel system combines deep learning-based detection with physics-based tracking to overcome the limitations of existing approaches. Our contributions include: (1) a comprehensive system design for object detection and tracking of fast-moving small objects in 3D space, (2) an innovative physics-based tracking algorithm that integrates kinematics motion equations to handle outliers and missed detections, and (3) an outlier detection and correction module that significantly improves tracking performance in challenging scenarios such as occlusions and rapid direction changes. We evaluated our proposed system on a custom racquetball dataset. Our evaluation shows our system surpassing kalman filter based trackers with up to 70\% less Average Displacement Error. Our system has significant applications for improving robot perception on autonomous platforms and demonstrates the effectiveness of combining physics-based models with deep learning approaches for real-time 3D detection and tracking of challenging small objects.

Figures (6)

• Figure 1: Illustration of our target object in an ideal racquetball game. Row 1 shows the ball as seen from a camera approximately 35 feet away from the front wall and Row 2 shows a zoomed-in picture of our target object. Our proposed approach improves tracking of the tiny ball.
• Figure 2: System Design for ball detection, tracking, and forecasting. The design shows the end-to-end process of our 3D perception system. The highlighted parts of the design represent different sub-processes or modules. We are using an NVIDIA Jetson TX2 as our main computing device for both collecting the dataset and extracting the 3D position of the ball.
• Figure 3: Outline of the physics-based tracker.
• Figure 4: Sample of our RGB-D dataset. The figure shows the RGB image and its corresponding depth image. We used the RGB frame and its matching depth image to design our 3D tracking and trajectory forecasting system.
• Figure 5: Comparison of trajectories between object detection method, ground truth and our physics model. In both of scenarios our model can interpolate these gaps, using kinematics motion model, to produce continuous trajectory. The tiny rectangular boxes show interpolated positions.