4DSegStreamer: Streaming 4D Panoptic Segmentation via Dual Threads

TL;DR

This work tackles real-time streaming 4D panoptic segmentation in dynamic environments where latency must be minimized. It proposes 4DSegStreamer, a dual-thread framework that couples a predictive memory thread with an inference thread to maintain spatial-temporal memories and support fast, memory-guided predictions for incoming frames. Key contributions include a memory update mechanism with ego-pose forecasting and dynamic object alignment, plus an inverse forward flow iteration strategy for efficient memory querying. Extensive experiments on SemanticKITTI, nuScenes, and HOI4D demonstrate substantial accuracy-latency gains and robust performance under high FPS, showing strong generalization across outdoor and indoor scenes. The approach enables real-time, fine-grained perception suitable for autonomous driving and robotics, by richly integrating memory-based reasoning with fast inference paths.

Abstract

4D panoptic segmentation in a streaming setting is critical for highly dynamic environments, such as evacuating dense crowds and autonomous driving in complex scenarios, where real-time, fine-grained perception within a constrained time budget is essential. In this paper, we introduce 4DSegStreamer, a novel framework that employs a Dual-Thread System to efficiently process streaming frames. The framework is general and can be seamlessly integrated into existing 3D and 4D segmentation methods to enable real-time capability. It also demonstrates superior robustness compared to existing streaming perception approaches, particularly under high FPS conditions. The system consists of a predictive thread and an inference thread. The predictive thread leverages historical motion and geometric information to extract features and forecast future dynamics. The inference thread ensures timely prediction for incoming frames by aligning with the latest memory and compensating for ego-motion and dynamic object movements. We evaluate 4DSegStreamer on the indoor HOI4D dataset and the outdoor SemanticKITTI and nuScenes datasets. Comprehensive experiments demonstrate the effectiveness of our approach, particularly in accurately predicting dynamic objects in complex scenes.

Figures (5)

• Figure 1: Comparison of streaming performance at different FPS settings on the SemanticKITTI dataset. Our 4DSegStreamer demonstrates significant performance gains and exhibits a slower performance decline as the FPS increases, indicating its robustness as a more advanced 4D streaming system for panoptic segmentation tasks, particularly in high-FPS scenarios.
• Figure 2: 4DSegStreamer: The dual-thread system consists of a predictive thread and an inference thread, enabling real-time query for unseen future frames. The predictive thread updates the geometric and motion memories with the latest extracted feature and leverages the historical information to forecast future dynamics. The inference thread retrieves per-point predictions by geometrically aligning them with the current memory using ego-pose and dynamic object alignment. Here, mem$_i$ denotes the memory updated with the latest key frame f$_i$, while f${_{i:j}}$ represents incoming frame${_{i, i+1, ..., j}}$.
• Figure 3: Point-level and voxel-level methods in inference thread: orange points indicate the extracted features corresponding keyframe points, while blue points indicate the aligned incoming frame points querying the features from memory.
• Figure 4: Ego-pose Alignment and Dynamic Object Alignment: The green points represent the previously processed frame that has been used to update the memories and the blue points are the current querying frame. The yellow box highlights static objects that can be aligned through ego-pose alignment. The red box indicates dynamic objects, which require dynamic object alignment to achieve proper alignment.
• Figure S1: Streaming Perception Setting: Green points denote dynamic objects from the processed frame, whereas blue points represent the current frame at the time of prediction generated by the algorithm.