FASTer: Focal Token Acquiring-and-Scaling Transformer for Long-term 3D Object Detection

TL;DR

FASTer addresses the exponential growth in storage and computation in long-term LiDAR-based 3D detection by introducing focal tokens guided by Adaptive Scaling, which compresses point sequences while preserving salient geometry. It further employs a memory-efficient framework that stores only focal points and uses a Grouped Hierarchical Fusion strategy with a dual-layer decoder to integrate spatial and temporal information across frames. The key contributions are the focal-token representation with adaptive acquisition, a memory-efficient temporal encoding scheme, and the grouped hierarchical fusion paradigm that enables scalable long-term fusion with improved robustness. Empirical results on the Waymo Open Dataset show state-of-the-art performance and favorable efficiency, highlighting FASTer’s practical potential for real-time, long-span autonomous driving perception. The approach reduces a potential $O(FN)$ complexity to roughly $O(N)$, enabling longer temporal reasoning without prohibitive overhead.

Abstract

Recent top-performing temporal 3D detectors based on Lidars have increasingly adopted region-based paradigms. They first generate coarse proposals, followed by encoding and fusing regional features. However, indiscriminate sampling and fusion often overlook the varying contributions of individual points and lead to exponentially increased complexity as the number of input frames grows. Moreover, arbitrary result-level concatenation limits the global information extraction. In this paper, we propose a Focal Token Acquring-and-Scaling Transformer (FASTer), which dynamically selects focal tokens and condenses token sequences in an adaptive and lightweight manner. Emphasizing the contribution of individual tokens, we propose a simple but effective Adaptive Scaling mechanism to capture geometric contexts while sifting out focal points. Adaptively storing and processing only focal points in historical frames dramatically reduces the overall complexity. Furthermore, a novel Grouped Hierarchical Fusion strategy is proposed, progressively performing sequence scaling and Intra-Group Fusion operations to facilitate the exchange of global spatial and temporal information. Experiments on the Waymo Open Dataset demonstrate that our FASTer significantly outperforms other state-of-the-art detectors in both performance and efficiency while also exhibiting improved flexibility and robustness. The code is available at https://github.com/MSunDYY/FASTer.git.

Figures (6)

• Figure 1: Comparison of different temporal-based schemas. (a) Previous methods store and collect points from the complete historical point cloud, and concatenate them on the channels after sequential spatial and temporal fusions. (b) Our method dynamically obtains, stores and encodes notable points, condenses the longer sequences into one long sequence through the integrated Spatial-Temporal Fusion module.
• Figure 2: The overall architecture of FASTer. We begin by utilizing a region proposal network (RPN) to generate proposals. In the Single-frame Sequence Processing (SFP) illustrated in \ref{['sec:CFE']}, we randomly sample a sufficient number of points and apply adaptive scaling layers to extract geometric features while concurrently select notable points to store in the memory bank. In the Multi-frame Sequence Processing (MSP) illustrated in \ref{['sec:MFF']}, we just sample a reduced number of points and design a hierarchical fusion strategy to progressively condense the long temporal sequences into one single token sequence. Finally, a dual-layer decoder is introduced to aggregate the outputs of SSP and MSP. The blue and red cubes represent current and historical tokens respectively.
• Figure 3: The architecture of Ad-MHSA, adaptively compresses long sequence while preserving the most valuable information.
• Figure 4: The architecture of Intra-Group Fusion, taking $T'$ sequences as input and combining them to output a single sequence.
• Figure 5: Sensitivity Comparison of several region-based methods with respect to the points sampling ratio $\gamma$.