TimePillars: Temporally-Recurrent 3D LiDAR Object Detection

TL;DR

TimePillars tackles the challenge of long-range 3D LiDAR object detection under sparse single-frame data by introducing a temporally recurrent, hardware-friendly pipeline built on pillar-based BEV representations. It uses a convGRU memory after the backbone and a convolutional ego-motion compensation mechanism guided by an auxiliary task, enabling robust fusion of past frames without heavy 3D or sparse-convolution operations. The approach achieves superior long-range performance, particularly for cyclists and pedestrians, while maintaining real-time latency and outperforming single-frame and naive multi-frame baselines on the Zenseact Open Dataset. Ablation studies confirm the value of placing the memory after the backbone and of auxiliary supervision, establishing TimePillars as a practical and effective baseline for temporally-aware 3D LiDAR detection with hardware constraints.

Abstract

Object detection applied to LiDAR point clouds is a relevant task in robotics, and particularly in autonomous driving. Single frame methods, predominant in the field, exploit information from individual sensor scans. Recent approaches achieve good performance, at relatively low inference time. Nevertheless, given the inherent high sparsity of LiDAR data, these methods struggle in long-range detection (e.g. 200m) which we deem to be critical in achieving safe automation. Aggregating multiple scans not only leads to a denser point cloud representation, but it also brings time-awareness to the system, and provides information about how the environment is changing. Solutions of this kind, however, are often highly problem-specific, demand careful data processing, and tend not to fulfil runtime requirements. In this context we propose TimePillars, a temporally-recurrent object detection pipeline which leverages the pillar representation of LiDAR data across time, respecting hardware integration efficiency constraints, and exploiting the diversity and long-range information of the novel Zenseact Open Dataset (ZOD). Through experimentation, we prove the benefits of having recurrency, and show how basic building blocks are enough to achieve robust and efficient results.

Figures (5)

• Figure 1: Detection range from LiDAR point cloud data. Depicted in red, relevant established datasets KittiCaesar2019NuScenes:Drivingwaymo, which present annotations up to at most 75 m. In green, long-range information provided by Alibeigi2023ZenseactDriving, reaching around 200 m of distance.
• Figure 2: TimePillars pipeline overview. Our method takes LiDAR point clouds as input, and outputs object detection predictions based on past experience (memory) and observations in the present. Preprocessing and postprocessing stages accomodate input raw LiDAR data and detection predictions for feature encoding and bounding boxes extraction, respectively. Valuable information, early encoded as pillars and further developed by a 2D convolutional backbone, is saved by a convGRU-based memory unit. A feature EGO-motion compensation module, guided by auxiliary learning, guarantees memory correctness across time.
• Figure 3: Illustration of both the transformation module and its auxiliary task. A single convolutional layer maps the transformation of the hidden state, provided analytical transformation details in the channel dimension. A lightweight CNN auxiliary task ensures the validity of the transformation module by enforcing transformed features in the output.
• Figure 4: Illustration of a simplified training step. The hidden state is built by running the network for a fixed number of iterations with scans prior to the core frame, ignoring the output. Then, we perform a final pass through the network with the core frame (annotated) as input, and backpropagate to update the weights.
• Figure 5: BEV visualization of TimePillars conv-based applied to the driving context. Model predictions are depicted together with the corresponding ground truths. Vehicles in blue, cyclists in red and pedestrians in green. Note the heading direction indicated with an arrow.