CaRLi-V: Camera-RADAR-LiDAR Point-Wise 3D Velocity Estimation

TL;DR

CaRLi-V tackles dense point-wise 3D velocity estimation in dynamic environments by fusing RADAR, LiDAR, and camera data. It introduces the velocity cube, a dense RADAR-based representation of radial velocity, and combines it with dense optical flow and LiDAR range through a closed-form fusion to yield per-point velocity vectors. The approach enables distinguishing motion of different non-rigid agents and handling multiple moving objects, demonstrated on a custom dataset and implemented as an open-source ROS2 package. Limitations include radar angular resolution causing velocity bleeding and room for improvement via temporal filtering and improved Doppler context; the work offers a practical pathway toward robust motion understanding for robotic systems.

Abstract

Accurate point-wise velocity estimation in 3D is crucial for robot interaction with non-rigid, dynamic agents, such as humans, enabling robust performance in path planning, collision avoidance, and object manipulation in dynamic environments. To this end, this paper proposes a novel RADAR, LiDAR, and camera fusion pipeline for point-wise 3D velocity estimation named CaRLi-V. This pipeline leverages raw RADAR measurements to create a novel RADAR representation, the velocity cube, which densely represents radial velocities within the RADAR's field-of-view. By combining the velocity cube for radial velocity extraction, optical flow for tangential velocity estimation, and LiDAR for point-wise range measurements through a closed-form solution, our approach can produce 3D velocity estimates for a dense array of points. Developed as an open-source ROS2 package, CaRLi-V has been field-tested against a custom dataset and proven to produce low velocity error metrics relative to ground truth, enabling point-wise velocity estimation for robotic applications.

Figures (4)

• Figure 1: Resulting point clouds augmented with full velocity vectors displayed in red. The pipeline is able to (a) discern velocities between different dynamic agents; (b) estimate both radial and tangential velocities, as well as combinations of both, and (c, d) extract velocities from single parts of non-rigid moving agents, such as individual limbs in humans.
• Figure 2: The CaRLi-V pipeline is divided into three steps: RADAR preprocessing, where raw ADC RADAR data is used to compute the RADAR velocity cube, a dense representation of velocities in the environment; Camera preprocessing, where two consecutive camera images are used to compute optical flow vectors for each pixel in the image; Sensor fusion, where the LiDAR point cloud is projected into both representations to extract radial velocity and optical flow readings, with both estimates combined through a closed-form solution.
• Figure 3: Effect of thresholding on the velocity cube. Thresholding removes salt-and-pepper noise and improves spatial precision along the angular dimensions, concentrating velocity readings to regions corresponding to moving agents.
• Figure 4: Plots of the magnitude of the ground truth and estimated velocity vectors, as well as their decompositions into radial and tangential components. These are taken from Scene 1 of the provided dataset. Gray regions show areas where the target object moved out sensor range. Note the jitter in the ground truth is due to the fact that we use instantaneous velocity estimates of the centroid.